Mailing List Archive

On 10/12/06, Christian Theune <ct@gocept.com> wrote:
> Hi,
>
> I think DNS is still (or again?) fishy.
>
> Currently cvs.zope.org resolves to .171 for me (which should be 173).
> That's what at least on of the community DNS servers tells me. Other
> community DNS servers seem not to know anything about zope.org at all.
>
> Some protocols:
>
> ctheune@uter ~ $ host svn.zope.org
> svn.zope.org is an alias for cvs.zope.org.
> cvs.zope.org has address 63.240.213.171
>
> ctheune@uter ~ $ dig zope.org
> ...
> zope.org. 5739 IN NS ns1.zoneedit.com.
> zope.org. 5739 IN NS ns1.dataflake.org.
> zope.org. 5739 IN NS ns7.zoneedit.com.
> zope.org. 5739 IN NS cabana.palladion.com.
> zope.org. 5739 IN NS seconly.rackspace.com.
> zope.org. 5739 IN NS ns.qutang.net.
> ...
>

This is wrong, most of these slaves never coordinated with me to
receive a copy of the zone. only ns.qutang.net has a copy.

ns*.zope.com have semi-identical copies, but have not transferred the
latest zone from zoneedit afaik.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 08:03, Justizin wrote:
> This is wrong, most of these slaves never coordinated with me to
> receive a copy of the zone. only ns.qutang.net has a copy.
>
> ns*.zope.com have semi-identical copies, but have not transferred the
> latest zone from zoneedit afaik.

What do you mean "never coordinated with you"? I never even got
notified that my server is indeed on the list, and unless this works
like "normal" DNS, how to manually get the zone.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLjikRAx5nvEhZLIRAq77AJ9VUvBpOvkWs8uZMb3q2E/dEh6PogCcCXJ1
Wnl0xQ5xzG/uiwStJa0e+pQ=
=ROu7
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 08:44, Jens Vagelpohl wrote:

> On 12 Oct 2006, at 08:03, Justizin wrote:
>> This is wrong, most of these slaves never coordinated with me to
>> receive a copy of the zone. only ns.qutang.net has a copy.
>>
>> ns*.zope.com have semi-identical copies, but have not transferred the
>> latest zone from zoneedit afaik.
>
> What do you mean "never coordinated with you"? I never even got
> notified that my server is indeed on the list, and unless this
> works like "normal" DNS, how to manually get the zone.

I have set my side up as a slave and it pulled the data from
zoneedit. Again, notification to all those people whose servers are
on the list would probably be a good thing.

Does zoneedit notify slaves automatically? Or does it require manual
pulling-down of master data?

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLjonRAx5nvEhZLIRAtohAJ4zNijtyM0cYberMAjU/BF4CpLNOACgnAcQ
CtpKIF7wsArfw4UtcaA3p3k=
=tpQu
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Jens Vagelpohl <jens@dataflake.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 12 Oct 2006, at 08:03, Justizin wrote:
> > This is wrong, most of these slaves never coordinated with me to
> > receive a copy of the zone. only ns.qutang.net has a copy.
> >
> > ns*.zope.com have semi-identical copies, but have not transferred the
> > latest zone from zoneedit afaik.
>
> What do you mean "never coordinated with you"? I never even got
> notified that my server is indeed on the list, and unless this works
> like "normal" DNS, how to manually get the zone.
>

"normal" is subjective here. Of course, by "normal", you mean BIND-centric.

In any case, ZoneEdit does not send NOTIFY requests. I tried to start
a thread on this last tuesday or so and received no replies. Time
marches on.

Anyway, everything except these hosts need to be removed from the rotation:

ns1.zoneedit.com
ns7.zoneedit.com
ns.qutang.net
ns*.zope.com

I'd love to see more backups once they have copies of the zone. If
you want to grab a copy of the zone, you'll have to transfer manually
from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP
addresses:

64.34.177.88
69.20.0.180
8.7.96.28
70.84.6.50
63.240.213.250
70.168.181.3

I offered last week to try and set up a NOTIFY mechanism from my own
system based on an hourly cronjob, but there was no interest, so I
decided not to prioritize it. I'll have to write a couple of scripts
for this, so it's not going to happen overnight anyway.

Three nameservers is fine for now. Eight would be far better.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

> > On 12 Oct 2006, at 08:03, Justizin wrote:
> > > This is wrong, most of these slaves never coordinated with me to
> > > receive a copy of the zone. only ns.qutang.net has a copy.
> > >

And this is my fault because ZoneEdit has these hosts listed as NS
records. I've removed them until they grab copies of the zone. Root
nameservers should not be looking to these NS records afaik, because
they are authoritative for NS queries based on their own records.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 08:57, Justizin wrote:
> Anyway, everything except these hosts need to be removed from the
> rotation:
>
> ns1.zoneedit.com
> ns7.zoneedit.com
> ns.qutang.net
> ns*.zope.com

Then I suggest you do that and end the current confusion in regards
to which server does what (and which server even has the correct data).


> I'd love to see more backups once they have copies of the zone. If
> you want to grab a copy of the zone, you'll have to transfer manually
> from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP
> addresses:

No you don't. Setting a machine up as a slave, in that terrible bind-
centric world, will cause it to pull the data automatically.


> Three nameservers is fine for now. Eight would be far better.

I still don't understand why we would need that many... but I don't
want to discuss this any further. Matter of fact, since zoneedit does
not support NOTIFY it is probably a bad thing to even have my server
on the list. I suggest you limit the official servers to the ones you
mentioned, the zoneedit/qutang/zope.com hosts until NOTIFY is working.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLj2KRAx5nvEhZLIRAvd9AJ9kNgz+oq14VqEW9AZsyHrirQmcbgCfRyTm
DrDo2moe+MzVKW1XNIeGXsI=
=TgXr
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Jens Vagelpohl <jens@dataflake.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 12 Oct 2006, at 08:57, Justizin wrote:
> > Anyway, everything except these hosts need to be removed from the
> > rotation:
> >
> > ns1.zoneedit.com
> > ns7.zoneedit.com
> > ns.qutang.net
> > ns*.zope.com
>
> Then I suggest you do that and end the current confusion in regards
> to which server does what (and which server even has the correct data).
>

(a) I don't control the actual registrar records

(b) Yes, these were listed in the zone itself as the NS, but noone
should be doing lookups via these servers, because ZoneEdit is not
authoritative for the NS records of this zone, the registrar is.

I've removed them, but I politely request that you stop being an
asshole unless you want to wear this hat yourself.

I'm sick, I was stranded in the middle of nowhere when this change
took place, and I was rushed.

It's all of our fault. Don't make me come over there.

>
> > I'd love to see more backups once they have copies of the zone. If
> > you want to grab a copy of the zone, you'll have to transfer manually
> > from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP
> > addresses:
>
> No you don't. Setting a machine up as a slave, in that terrible bind-
> centric world, will cause it to pull the data automatically.
>

ZoneEdit apparently does not run BIND, or at least does not send
NOTIFY requests.

I don't know what you want me to do.

>
> > Three nameservers is fine for now. Eight would be far better.
>
> I still don't understand why we would need that many... but I don't
> want to discuss this any further. Matter of fact, since zoneedit does
> not support NOTIFY it is probably a bad thing to even have my server
> on the list. I suggest you limit the official servers to the ones you
> mentioned, the zoneedit/qutang/zope.com hosts until NOTIFY is working.
>
> jens
>

You don't understand because you're an idiot, Jens, and you've never
guaranteed 100% uptime.

I was basically shut up by your whining when I tried to explain all of
the precautions we should take in order to avoid what happened to
zope.org this week.

I won't respond to demands that I rush ever again.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

Justizin wrote:
> I'd love to see more backups once they have copies of the zone.

Why? zope.org has happily lived off two nameservers for years and years...

All of a sudden, we "need" to have more backups, the upshot of which has
been people in europe getting served bad dns from ns.qutang.net :-(

What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com
(could we also use ns(2-6).zoneedit.com?) and be done with it?

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 09:15, Justizin wrote:
> (a) I don't control the actual registrar records
>
> (b) Yes, these were listed in the zone itself as the NS, but noone
> should be doing lookups via these servers, because ZoneEdit is not
> authoritative for the NS records of this zone, the registrar is.

To stay strictly on technical issues, I think you're constantly
implying that the DNS servers for the zope.org zone that are listed
by the registrar are not the same as the DNS servers the zone data
itself contains. Can you explain why this discrepancy exists, or why
it makes sense?


>> > I'd love to see more backups once they have copies of the zone. If
>> > you want to grab a copy of the zone, you'll have to transfer
>> manually
>> > from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP
>> > addresses:
>>
>> No you don't. Setting a machine up as a slave, in that terrible bind-
>> centric world, will cause it to pull the data automatically.
>>
>
> ZoneEdit apparently does not run BIND, or at least does not send
> NOTIFY requests.
>
> I don't know what you want me to do.

Nothing. I am describing the situation where you have a bind slave
and you are configuring a slave zone for the first time. At that
moment you don't have to manually pull the zone data, bind will
magically fetch it. This was a hint for people who might want to set
up a slave.

jens



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLkJnRAx5nvEhZLIRApZWAKCdD4MxCtrJuZ+ezihcYnnC+KugmQCghgEC
bAxQ9hjKbWdXHVdz5nuTzT8=
=0e5C
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Chris Withers <chris@simplistix.co.uk> wrote:
> Justizin wrote:
> > I'd love to see more backups once they have copies of the zone.
>
> Why? zope.org has happily lived off two nameservers for years and years...
>
> All of a sudden, we "need" to have more backups, the upshot of which has
> been people in europe getting served bad dns from ns.qutang.net :-(

This is a logical fallacy. Services were not unavailable because we
have more than two nameservers, services were unavailable because we
rushed.

ns.qutang.net did not serve any bad dns that ns*.zoneedit.com were not
serving. The errors were in ZoneEdit's copy of the Zone.

I was thinking just now over a smoke about someone I used to work with
at Rackspace, the datacenter engineer. Bob was a member of the NASA
Challenge Safety Team. He personally recommended against launching
the Challenger, which exploded, killing some astronauts.

I learned from working with him that you should never tell someone
with more experience to be less cautious.

> What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com
> (could we also use ns(2-6).zoneedit.com?) and be done with it?

We can only use the nameservers that zoneedit allocates us.

Yanno, people used to pay $75 per half hour for this expertise.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 09:20, Chris Withers wrote:

> Justizin wrote:
>> I'd love to see more backups once they have copies of the zone.
>
> Why? zope.org has happily lived off two nameservers for years and
> years...
>
> All of a sudden, we "need" to have more backups, the upshot of
> which has been people in europe getting served bad dns from
> ns.qutang.net :-(
>
> What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com
> (could we also use ns(2-6).zoneedit.com?) and be done with it?

It makes sense to have name servers in different physical locations
and on different networks in case one provider runs into trouble. The
point of contention is the number of slaves.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLkLoRAx5nvEhZLIRArSuAKC1xDSZzd+Y4elgChwKb8i9INCerACfZMBZ
wdI8SlUIRqp+QWM6Wbj7wqw=
=zPH2
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

Jens Vagelpohl wrote:
> It makes sense to have name servers in different physical locations and
> on different networks in case one provider runs into trouble. The point
> of contention is the number of slaves.

Right, which brings me back to my other point: why, when 2 server have
been fine for about a decade, do we need to change now?

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

>
> Yanno, people used to pay $75 per half hour for this expertise.
>

.. and I am charging $3,000 for a server move / consolidation in the
range of what zope.org wants to see happen in the next few months.

Sometimes, even paying clients insist on the wrong approach, or think
that I am overcomplicating things. That's why I require a large
portion of payment up front. If someone wants to fire me before I
have done much work and after I have received $1,500, that's always
acceptable.

Anyway, look, here's the deal - stop inferring that my suggested
precautions are stupid, or zope.org will go down again soon, probably,
and it will be YOUR fault, and I will laugh at you. As it's MY fault
this time, I'm going to tell you, it doesn't feel great. I approached
this change with nothing less than the expectation that I would like
to be able to continue reaching zope.org on a daily basis, and it went
badly. I could not be less pleased.

Insofar as anyone who wants to flog me, look - if www.siggraph.org is
unreacheable, I get flogged by the ACM SIGGRAPH President, because he
uses it in his Human Computer Interface courses, and it makes him look
like an asshole.

Truth be told, I care about zope.org working for my own purposes a
great deal more than siggraph.org. So, I wasn't intentionally sloppy.

Yes, I could write a fifty step DNS migration tutorial, and I could
point out a few steps that I skipped.

The fact is, I would spend a month planning a DNS move if it were up to me.

So let's all stop pointing fingers and move on. I just audited the
DNS config, there was another small mistake for the secondary
nameserver, which pointed at ns2.zope.org rather than ns2.zope.com,
it's fixed. As long as mail.zope.org does not go down in the next
couple of days, that should not cause any perceivable problems.

Let's move on. What in the heck do we want to do about apache? If we
want a dedicated environment for _just_ flat files served by apache,
and not zope, I might suggest looking at a VPS. I know I can have one
set up for about $20 with reasonable specs for running nothing but
apache.

Heck, I can take that out of pocket, esp if the Zope Foundation is 501(3)c.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

Just a couple of notes here.

Although zoneedit has been running fine for me for years without a
single problem, obviously it would be nice with some backup.
Preferably something with another ISP and located on like another
continent or something. Two of these backups would be even better.

But honestly, compare the likelyhood that all three of these would
fail at one time, together with the increasing likelyhood than one
server of them is misconfigured and starts disturbing the usage for a
minor part of the users, then we will quickly realize that the more
backups and failsafes we have the larger the likelyhood that something
of this will go wrong.

8 servers seems to be to be a complete overkill, and it will only
cause problems. I will change my mind on this the time all zone-edit
servers stop working at the same time as two of the backups fail.

Don't overcomplicate things. It just makes them fail.
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Jens Vagelpohl <jens@dataflake.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 12 Oct 2006, at 09:15, Justizin wrote:
> > (a) I don't control the actual registrar records
> >
> > (b) Yes, these were listed in the zone itself as the NS, but noone
> > should be doing lookups via these servers, because ZoneEdit is not
> > authoritative for the NS records of this zone, the registrar is.
>
> To stay strictly on technical issues, I think you're constantly
> implying that the DNS servers for the zope.org zone that are listed
> by the registrar are not the same as the DNS servers the zone data
> itself contains. Can you explain why this discrepancy exists, or why
> it makes sense?
>

I prepared a copy of the zone in ZoneEdit with small changes to
reflect the plans for a new configuration, including new nameservers.

I pulled the zone into ns.qutang.net early last week and sent out an
e-mail which, surely, was just lost in the white noise. oh well.

so, because we wanted to start modifying the zone really soon, i told
rob page to change the registrar to point at:

ns1.zoneedit.com
ns7.zoneedit.com
ns.qutang.net

These nameservers all had the same data, including the same incorrect
records. FWIW, three records with the same IP address went sour:

www.zope.org
cvs.zope.org
zope.org

This is curious, because I recall making an effort to individually
copy each record from the zone file that Rob sent me, to avoid just
this sort of mistake.

whatever, these records pointed at .1 instead of .171

>
> Nothing. I am describing the situation where you have a bind slave
> and you are configuring a slave zone for the first time. At that
> moment you don't have to manually pull the zone data, bind will
> magically fetch it. This was a hint for people who might want to set
> up a slave.
>

Handy.

I am writing a how-to for making djbdns comply with both ends of the
NOTIFY chain. There are a bunch of tools for this, very simple
djb-ish stuff, but nothing is part of the package.

If someone running BIND wants to pull from zoneedit and send the rest
of us NOTIFY requests when a change is detected, we can pretty much do
that now. I should be set up to respond to NOTIFY. I have to add
something into the tinydns-data chain which enacts changes to live
configuration so that it spurs a NOTIFY to slaves.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Lennart Regebro <regebro@gmail.com> wrote:
> Just a couple of notes here.
>
> Although zoneedit has been running fine for me for years without a
> single problem, obviously it would be nice with some backup.
> Preferably something with another ISP and located on like another
> continent or something. Two of these backups would be even better.
>
> But honestly, compare the likelyhood that all three of these would
> fail at one time, together with the increasing likelyhood than one
> server of them is misconfigured and starts disturbing the usage for a
> minor part of the users, then we will quickly realize that the more
> backups and failsafes we have the larger the likelyhood that something
> of this will go wrong.

the worst that happens is that some changes fail to propogate.
changes to DNS should always be approached with the assumption that
this will happen. What's worse is for there to be no copy of a zone
available.

It should never be necessary for an A record to change immediately,
because this cannot be relied upon. The best defense to this is,
however, to set TTLs at 300s, or 5 minutes, about a week in advance.

> 8 servers seems to be to be a complete overkill, and it will only
> cause problems. I will change my mind on this the time all zone-edit
> servers stop working at the same time as two of the backups fail.

It could cause problems, and that's why we aren't really using eight
servers right now, but it should not cause problems. It is a
challenge, also, that our DNS is not hosted in the same location as
the website. So, it's possible that DNS will be unreachable when an
outage occurs, i.e. a fibre being cut in the middle of the ocean, and
this outage may not actually affect our site.

I bet ten bucks if we rely entirely on zoneedit's nameservers that
this will happen once for at least twelve hours for some significant
region of the world within the next year.

> Don't overcomplicate things. It just makes them fail.

This assumption really has nothing to do with what happened this week.

What happened this week was either:

(a) a typo

(b) an erroneously truncated string

If there were only two nameservers, they would have pointed at the
wrong IP, and the site would have been perceptually unavailable for a
few hours to two days for various people. If there were eight, the
same would happen, for about the same time frame.

So, if you want to only use two nameservers, that's okay with me.
Remember to wake me up when the zone is unreachable for someone and we
want to run more. :)

I always assume, if anything, that some machines, network connections,
disk drives, etc.. will invariably fail, and that you can never have
too many if they are available. I like the idea of a group of zope
community members collectively providing DNS service. Maybe we should
even talk about running multiple copies of the flat content in
different places. If my site goes down, esp if one of my machines
fail, I much prefer to feel comfortable that I can reach zope.org than
rely on the possibility that i might have copies of recent releases in
another location. if i'm going to keep copies of the releases around
for myself, might as well mirror them, eh?

While having a set of servers configured by various people sounds as
if it would be overcomplicated, with proper planning and coordination,
we should be able to keep it simple.

When making changes to DNS, always assume that for 48 hours there will
be between a 90-10 and 10-90 split between people who have your new
records and people who have old records. When changing nameservers,
double or triple this, because some people will have cached records
from the old nameserver *and* more recently cached NS records, so they
may continue querying the old nameserver until the cached NS record
itself expires.

When something critical like svn/cvs or the main website need to be
changed, again, it is necessary to drop the TTL, on the entire zone,
even, to something really short like 300s about a week in advance.
This ensures that everyone in the world has a copy of the zone which
says: "no copy of this zone and no records in this zone are good for
longer than five minutes.". Just before a switch is made, you can
proxy the old front-end apache server to the new host explicitly, and
then update records. for five or ten minutes some people's requests
will be slow because they are possibly doubling-back across the
internet, but at least they can't really tell what's going on, just
that for a few minutes it is a 'little bit slow'.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Justizin <justizin@siggraph.org> wrote:
> It could cause problems, and that's why we aren't really using eight
> servers right now, but it should not cause problems.

Servers should not fail. This should not cause problems. But in
reality, it will.

> It is a
> challenge, also, that our DNS is not hosted in the same location as
> the website. So, it's possible that DNS will be unreachable when an
> outage occurs, i.e. a fibre being cut in the middle of the ocean, and
> this outage may not actually affect our site.

Which is why one or two backups on another continent is nice to have.

> > Don't overcomplicate things. It just makes them fail.
>
> This assumption really has nothing to do with what happened this week.

I'm not convinced.

> So, if you want to only use two nameservers, that's okay with me.

Please respons to what I write, and argue against what I argue,
instead of making up arguments against things I have never said. I,
explicitly in my last mail, said that one or two backups on other
continents would be necssary, but that the previously mentioned
*eight* backups would cause more problems than they solve.

If you don't agree with this, you are welcome to explain to me why.
But do NOT argue against me by implying that I have said something
stupid, which I never said.

Thank you.

--
Lennart Regebro, Nuxeo http://www.nuxeo.com/
CPS Content Management http://www.nuxeo.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 10:05, Lennart Regebro wrote:
> But honestly, compare the likelyhood that all three of these would
> fail at one time, together with the increasing likelyhood than one
> server of them is misconfigured and starts disturbing the usage for a
> minor part of the users, then we will quickly realize that the more
> backups and failsafes we have the larger the likelyhood that something
> of this will go wrong.
>
> 8 servers seems to be to be a complete overkill, and it will only
> cause problems. I will change my mind on this the time all zone-edit
> servers stop working at the same time as two of the backups fail.
>
> Don't overcomplicate things. It just makes them fail.

Exactly.

We are not building a carrier-grade solution here because, as the
programmer idiom goes, it is YAGNI (you ain't gonna need it).

Keeping a carrier-grade solution running correctly is always more
effort than keeping the simple solution up. There's a diminishing
return between upkeep/effort/maintenance/script-writing and "oops,
DNS is gone for an hour". I seriously don't see the added value.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLmpZRAx5nvEhZLIRAt/JAKCtd4n0eXB+40oC9taJu9NXjzpsjQCgrxpt
EWr/MZcXHi7iMWqNkKNYdiU=
=OHbm
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Lennart Regebro <regebro@gmail.com> wrote:
> On 10/12/06, Justizin <justizin@siggraph.org> wrote:
> > It could cause problems, and that's why we aren't really using eight
> > servers right now, but it should not cause problems.
>
> Servers should not fail. This should not cause problems. But in
> reality, it will.
>

Servers failing will not cause problems, the only real risk would be tampering.

The reason for having many servers is to protect against failure.

> > It is a
> > challenge, also, that our DNS is not hosted in the same location as
> > the website. So, it's possible that DNS will be unreachable when an
> > outage occurs, i.e. a fibre being cut in the middle of the ocean, and
> > this outage may not actually affect our site.
>
> Which is why one or two backups on another continent is nice to have.
>

"Three or more" is best.

> > > Don't overcomplicate things. It just makes them fail.
> >
> > This assumption really has nothing to do with what happened this week.
>
> I'm not convinced.
>

Then take over, Lennart. I do not care.

You don't have to be convinced. Explain to me how this problem is
related to the outage, which was as simple as this:

records served by three of five nameservers were incorrect. the
other two were zope.com nameservers, and they don't delegate to
zoneedit afaik.

> > So, if you want to only use two nameservers, that's okay with me.
>
> Please respons to what I write, and argue against what I argue,
> instead of making up arguments against things I have never said. I,
> explicitly in my last mail, said that one or two backups on other
> continents would be necssary, but that the previously mentioned
> *eight* backups would cause more problems than they solve.

You said you don't understand why we don't just use zoneedit.

What makes four servers less failure prone than eight, so long as they
all agree that zoneedit is in charge.

> If you don't agree with this, you are welcome to explain to me why.
> But do NOT argue against me by implying that I have said something
> stupid, which I never said.

Oh whatever.

Look, I'm sick of this conversation. I did a better job than anyone
else in the conversation would have, and problems happened because we
spent a week on something that we should have spent 2-4 weeks on. We
learned something.

I think the real issue is that we ran into a problem, which I tried
hard to avoid, and people are still arguing that I am proposing to
take too many precautions.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Justizin <justizin@siggraph.org> wrote:
> Servers failing will not cause problems, the only real risk would be tampering.

I was unclear, sorry.

What I ment to say is that things go wrong. Your statement "this
should not cause problems", is equivalent to "servers will not fail"
and my point then was that in that case we can run with one server and
be done with it.

The reality is that servers fail. The reality is also that complex
setups cause problems, no matter that they "shouldn't".

> The reason for having many servers is to protect against failure.

With increasing number of servers you get better protection against
failure. But the increasing protection you get gets less and less
with each server. At the same time, configuration weirdness and other
stuff is likely to INCREASE the error rate the more backups you have,
because of Murphys law and other stuff.

At one point, this increase in problems will overwhelm the increase in
protection.

I would also like to claim that this crossover point is nowehere near
the previously mentioned number of eight servers, but rather closer
two have one or two backups on another continent.

Some maths:

Say that a server fails one day per month in average (which is way
more than we really will have). One backup server located on anotehr
continent then means that we will statistically have DNS outage only
one day in 900. Thats one day every three years. Two backups located
on different continents will give us a failure rate of one day per
27000 days. That's one day every seventy-fifth year.

How would five-six increasing backup servers in any reasonable way
actually increase that realiability? It wouldn't, because for every
server you add, you increase the risk of something going wrong. That's
probably not an exponential risk, but I'm pretty sure somebody
somewhere will fuck something up more often than every seventy-fifth
year, so I don't actually think that having more than two backups on
different continents is gonna increase realiability.

> "Three or more" is best.

If you talk about total number of DNS servers, then I agree.
Two at zoneedit, one or two more somewhere else.

> Then take over, Lennart. I do not care.

Oh, you do care, because you get angry-

> You said you don't understand why we don't just use zoneedit.

No. I have never said anything like that. Please read what I say, and
answer that. I have been discussing politics on the internet for 15
years, and one thing I have learned is to completely stop any
discussion when you get accused of an opinion you don't have because
constructive discussion have at that point failed.

Please read my emails, and answer they things I said, not the things I
did not say.

> What makes four servers less failure prone than eight, so long as they
> all agree that zoneedit is in charge.

I think that is a pretty obvious question. The more things you have
the more things will fail.

> Look, I'm sick of this conversation. I did a better job than anyone
> else in the conversation would have, and problems happened because we
> spent a week on something that we should have spent 2-4 weeks on. We
> learned something.

That is quite possible. I am not claiming you did a bad job. I have
never said I would do a better job. I don't complain, whine or say you
are stupid. I'm say one simple thing:

Having eight servers is overkill and cause more problems than it solves.

Please discuss this instead of trying to make this be about some sort
of personal issue. It is not. You are a professional. I am a
professional. Lets please all behave like it.

--
Lennart Regebro, Nuxeo http://www.nuxeo.com/
CPS Content Management http://www.nuxeo.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

FYI, there's a problem with your host Justizin:

> server ns1.zoneedit.com
Default server: ns1.zoneedit.com
Address: 207.234.248.200#53
> cvs.zope.org
Server: ns1.zoneedit.com
Address: 207.234.248.200#53

Name: cvs.zope.org
Address: 63.240.213.173
> server ns.qutang.net
Default server: ns.qutang.net
Address: 70.84.6.50#53
> cvs.zope.org
Server: ns.qutang.net
Address: 70.84.6.50#53

Name: cvs.zope.org
Address: 63.240.213.171
>


In my opinion, the registrar should only have zoneedit.com servers in it for
the time being.

Andrew


On 10/12/06 11:02 AM, "Justizin" <justizin@siggraph.org> wrote:

> On 10/12/06, Lennart Regebro <regebro@gmail.com> wrote:
>> Just a couple of notes here.
>>
>> Although zoneedit has been running fine for me for years without a
>> single problem, obviously it would be nice with some backup.
>> Preferably something with another ISP and located on like another
>> continent or something. Two of these backups would be even better.
>>
>> But honestly, compare the likelyhood that all three of these would
>> fail at one time, together with the increasing likelyhood than one
>> server of them is misconfigured and starts disturbing the usage for a
>> minor part of the users, then we will quickly realize that the more
>> backups and failsafes we have the larger the likelyhood that something
>> of this will go wrong.
>
> the worst that happens is that some changes fail to propogate.
> changes to DNS should always be approached with the assumption that
> this will happen. What's worse is for there to be no copy of a zone
> available.
>
> It should never be necessary for an A record to change immediately,
> because this cannot be relied upon. The best defense to this is,
> however, to set TTLs at 300s, or 5 minutes, about a week in advance.
>
>> 8 servers seems to be to be a complete overkill, and it will only
>> cause problems. I will change my mind on this the time all zone-edit
>> servers stop working at the same time as two of the backups fail.
>
> It could cause problems, and that's why we aren't really using eight
> servers right now, but it should not cause problems. It is a
> challenge, also, that our DNS is not hosted in the same location as
> the website. So, it's possible that DNS will be unreachable when an
> outage occurs, i.e. a fibre being cut in the middle of the ocean, and
> this outage may not actually affect our site.
>
> I bet ten bucks if we rely entirely on zoneedit's nameservers that
> this will happen once for at least twelve hours for some significant
> region of the world within the next year.
>
>> Don't overcomplicate things. It just makes them fail.
>
> This assumption really has nothing to do with what happened this week.
>
> What happened this week was either:
>
> (a) a typo
>
> (b) an erroneously truncated string
>
> If there were only two nameservers, they would have pointed at the
> wrong IP, and the site would have been perceptually unavailable for a
> few hours to two days for various people. If there were eight, the
> same would happen, for about the same time frame.
>
> So, if you want to only use two nameservers, that's okay with me.
> Remember to wake me up when the zone is unreachable for someone and we
> want to run more. :)
>
> I always assume, if anything, that some machines, network connections,
> disk drives, etc.. will invariably fail, and that you can never have
> too many if they are available. I like the idea of a group of zope
> community members collectively providing DNS service. Maybe we should
> even talk about running multiple copies of the flat content in
> different places. If my site goes down, esp if one of my machines
> fail, I much prefer to feel comfortable that I can reach zope.org than
> rely on the possibility that i might have copies of recent releases in
> another location. if i'm going to keep copies of the releases around
> for myself, might as well mirror them, eh?
>
> While having a set of servers configured by various people sounds as
> if it would be overcomplicated, with proper planning and coordination,
> we should be able to keep it simple.
>
> When making changes to DNS, always assume that for 48 hours there will
> be between a 90-10 and 10-90 split between people who have your new
> records and people who have old records. When changing nameservers,
> double or triple this, because some people will have cached records
> from the old nameserver *and* more recently cached NS records, so they
> may continue querying the old nameserver until the cached NS record
> itself expires.
>
> When something critical like svn/cvs or the main website need to be
> changed, again, it is necessary to drop the TTL, on the entire zone,
> even, to something really short like 300s about a week in advance.
> This ensures that everyone in the world has a copy of the zone which
> says: "no copy of this zone and no records in this zone are good for
> longer than five minutes.". Just before a switch is made, you can
> proxy the old front-end apache server to the new host explicitly, and
> then update records. for five or ten minutes some people's requests
> will be slow because they are possibly doubling-back across the
> internet, but at least they can't really tell what's going on, just
> that for a few minutes it is a 'little bit slow'.


_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

Can we have only zoneedit as the registered nameservers? 3 out of the 5
listed name servers at the registrar are wrong. We need this fixed ASAP.


Andrew


_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 12 Oct 2006, at 13:57, Andrew Sawyers wrote:

> Can we have only zoneedit as the registered nameservers? 3 out of
> the 5
> listed name servers at the registrar are wrong. We need this fixed
> ASAP.

Just to close this out, Rob has now changed the info at the registrar
to only show ns1.zoneedit.com and ns7.zoneedit.com. This change will
take a few hours to become visible. We can add more DNS servers when
this initial mess has rectified itself. Since the zoneedit
information is correct this is just a matter of time now.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFFLroyRAx5nvEhZLIRAu3SAJ9eXWx4jpyRlSN5pqVPKGuCs0ZmgQCfejX5
tpl1F9D62fMaXOF3zfHHq9o=
=yIS3
-----END PGP SIGNATURE-----
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web

On 10/12/06, Jens Vagelpohl <jens@dataflake.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 12 Oct 2006, at 10:05, Lennart Regebro wrote:
> > But honestly, compare the likelyhood that all three of these would
> > fail at one time, together with the increasing likelyhood than one
> > server of them is misconfigured and starts disturbing the usage for a
> > minor part of the users, then we will quickly realize that the more
> > backups and failsafes we have the larger the likelyhood that something
> > of this will go wrong.
> >
> > 8 servers seems to be to be a complete overkill, and it will only
> > cause problems. I will change my mind on this the time all zone-edit
> > servers stop working at the same time as two of the backups fail.
> >
> > Don't overcomplicate things. It just makes them fail.
>
> Exactly.
>
> We are not building a carrier-grade solution here because, as the
> programmer idiom goes, it is YAGNI (you ain't gonna need it).
>
> Keeping a carrier-grade solution running correctly is always more
> effort than keeping the simple solution up. There's a diminishing
> return between upkeep/effort/maintenance/script-writing and "oops,
> DNS is gone for an hour". I seriously don't see the added value.
>

It's not about "carrier-grade". That's a total misconception.

Carriers have big systems, we want lots of alternates in case one of
those big systems goes down.

That's my opinion.

--
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/
_______________________________________________
Zope-web maillist - Zope-web@zope.org
http://mail.zope.org/mailman/listinfo/zope-web