Some readers write in private mail:
> There's nothing I know that can contribute meaningfully on HTTPS. If
> there is unvaluable overhead involved, I would suggest going back to raw
> TCP. We have to parse the commands coming over HTTP or any of the
> non-object-savvy communications protocols above at any rate. The only
> advantage I can see to using HTTP over TCP is that firewalls are more
> likely to let it through. That advantage is virtually irrelevant in our
> scenarios though.
>
> Whichever of the above protocols will do the command parsing for us and
> invoke the appropriate code would suit me nicely. Perhaps I need to go
> find a list of them and learn the advantages/disadvantages of each.
I think you are too frightened by the parsing/reply overheard of http.
It really is pretty trivial. The point for me is that everybody knows
it and is comfortable dealing with it. Writing another protocol/message
format on tcp is novelty we should avoid, I think. The real race for
would seem to be amongst http, https, sun rpc, and corba, and even I'm
willing to write off corba. Given the complexity of doing sun rpc on
Windows, I'm personally inclined to think that http/https are the
best available choices. I'm waiting for someone to try to convince
me otherwise.
> > I am not convinced that we have adequately addressed the issue of which comes
> > first, group membership or GRITS command protocols. There seems to be a
> > (sometimes not so) subtle intermixture of the two notions.
OK, we will continue to strenghten the language: you get quorum, -then- you
can issue grits commands. You must obtain quorum w/o using grits-controlled
resources for that group. You might have -another- group that uses a
grits-controlled resource to resolve a quorum dispute. But I'm not sure
that doesn't recurse without terminating.
> > I think that the problem scenarios are not associated with joining, but with
> > leaving, i.e when members are supposed to leave but behave in a naughty
> > fashion.
We need to look into this some more, and come up with some scenarios.
A problem case that came up yesterday was the sole survivor who fences
everybody else out, then crashes gracelessly. How does a booting membership,
with no knowledge of the past state get to a point where it can issue commands
to the grits-locked resources? If they start at generation 0 again, live
resources could reject their commands as from a too-old generation. I suppose
if we trusted the "challenge" protocol from the resource that we could assert
that yes, even though the generation is before the last one, we really are
in control. If that works, then we don't need memory in the group side
of the generation either, at least for grits purposes.
> Okay. I will plead guilty on getting group membership and fencing
> intermingled a little more than appropriate. We may need to develop a
> separate quorum resource then for the overall project. The open
> standards community would be happy enough to have one, I suspect. Using
> GRITS to do one may be possible, too, but designing a clean one would be
> a little nicer.
I don't want that to become part of grits. I -really- want to
try to keep it separate. There are already existing membership
services that establish quorum, but do a bad job of fencing. So those,
by assertion, proove that you don't need grits to get quorum, and that
therefore grits may presume presence of quorum by the commanding cluster.
> Question: Does a project like this after the discussion is done ...end up
> with an RFC page?
Maybe, maybe not. From my experience writing the RFC 1697 RDBMSMIB, and
watching the SNMP extension wars, it is difficult to get an rfc for
things that aren't services on the public net. It is even more difficult
if they don't assume/require IP messaging! In as much as GRITS/Natalie
don't so far seem like public services, that is a strike. Natalie seems
closer, actually, since NFS is a public service, and there are RFCs for it.
Where GRITS is used to talk to fibrechannel switches, it's really a stretch.
> > > > > FIXME - For this to work, we need to fix the width of the
> > > > > generation, to 16, 32, or 64 bits. My inclination is to make it
> > > > > 64 bits.
> > > >
> > > > Putting it at 64 bits would make it rather unlikely that the ceiling
> > > > is ever hit. 1ms network latency * 2^64 is "sizable". Some machines
> > > > may have to do their own 64 bit arithmetic if they don't have 64
> > > > bit libraries (or 64 bit processors), but that's not a serious issue.
> > >
> > > Yes, but arguing against myself, I'm not sure I want to -require- 64
> > > bits, esp. if I'm trying to work with an existing group manager that
> > > has a smaller native width. GRITS/NATALIE is not supposed to be
> > > Linux specific, so I kind of don't like wiring things in like
> > > generation size.
> > >
> > > Any opinions?
> >
> > I think that on any platform double-word arihmetic is simple and it would be
> > best to avoid any potential wrap, although i think that testing should be done
> > with an 8 bit number to ensure that the wrap works.
> Fair enough. Put it as a typedef or #define in the initial
> implementation to at least test it, and then publish it switched to the
> 64 bit mode. 1 ms latency between machines (as a minimum time for
> reconfig duration) comes to 584 million years before we wrap. I know
> people in previous decades have hit y2k problems by considering that
> their code couldn't possibly be still in use for 20 or 30 years, but
> perhaps we're safe that our code will be long forgotten after 584
> million. :) Bit rot should take care of that nicely enough, if nothing
> else.
The point I'm trying to address is the pre-existing cluster manager that
uses a 16 bit generation counter, then wraps it itself. Do we -require-
it to change to 32 or 64 bits to use GRITS? That seems like a pretty
big demand, and we've been trying to be accomodating to existing systems.
This is irrelevant to the Linux guys, but important for other places.
-dB
> There's nothing I know that can contribute meaningfully on HTTPS. If
> there is unvaluable overhead involved, I would suggest going back to raw
> TCP. We have to parse the commands coming over HTTP or any of the
> non-object-savvy communications protocols above at any rate. The only
> advantage I can see to using HTTP over TCP is that firewalls are more
> likely to let it through. That advantage is virtually irrelevant in our
> scenarios though.
>
> Whichever of the above protocols will do the command parsing for us and
> invoke the appropriate code would suit me nicely. Perhaps I need to go
> find a list of them and learn the advantages/disadvantages of each.
I think you are too frightened by the parsing/reply overheard of http.
It really is pretty trivial. The point for me is that everybody knows
it and is comfortable dealing with it. Writing another protocol/message
format on tcp is novelty we should avoid, I think. The real race for
would seem to be amongst http, https, sun rpc, and corba, and even I'm
willing to write off corba. Given the complexity of doing sun rpc on
Windows, I'm personally inclined to think that http/https are the
best available choices. I'm waiting for someone to try to convince
me otherwise.
> > I am not convinced that we have adequately addressed the issue of which comes
> > first, group membership or GRITS command protocols. There seems to be a
> > (sometimes not so) subtle intermixture of the two notions.
OK, we will continue to strenghten the language: you get quorum, -then- you
can issue grits commands. You must obtain quorum w/o using grits-controlled
resources for that group. You might have -another- group that uses a
grits-controlled resource to resolve a quorum dispute. But I'm not sure
that doesn't recurse without terminating.
> > I think that the problem scenarios are not associated with joining, but with
> > leaving, i.e when members are supposed to leave but behave in a naughty
> > fashion.
We need to look into this some more, and come up with some scenarios.
A problem case that came up yesterday was the sole survivor who fences
everybody else out, then crashes gracelessly. How does a booting membership,
with no knowledge of the past state get to a point where it can issue commands
to the grits-locked resources? If they start at generation 0 again, live
resources could reject their commands as from a too-old generation. I suppose
if we trusted the "challenge" protocol from the resource that we could assert
that yes, even though the generation is before the last one, we really are
in control. If that works, then we don't need memory in the group side
of the generation either, at least for grits purposes.
> Okay. I will plead guilty on getting group membership and fencing
> intermingled a little more than appropriate. We may need to develop a
> separate quorum resource then for the overall project. The open
> standards community would be happy enough to have one, I suspect. Using
> GRITS to do one may be possible, too, but designing a clean one would be
> a little nicer.
I don't want that to become part of grits. I -really- want to
try to keep it separate. There are already existing membership
services that establish quorum, but do a bad job of fencing. So those,
by assertion, proove that you don't need grits to get quorum, and that
therefore grits may presume presence of quorum by the commanding cluster.
> Question: Does a project like this after the discussion is done ...end up
> with an RFC page?
Maybe, maybe not. From my experience writing the RFC 1697 RDBMSMIB, and
watching the SNMP extension wars, it is difficult to get an rfc for
things that aren't services on the public net. It is even more difficult
if they don't assume/require IP messaging! In as much as GRITS/Natalie
don't so far seem like public services, that is a strike. Natalie seems
closer, actually, since NFS is a public service, and there are RFCs for it.
Where GRITS is used to talk to fibrechannel switches, it's really a stretch.
> > > > > FIXME - For this to work, we need to fix the width of the
> > > > > generation, to 16, 32, or 64 bits. My inclination is to make it
> > > > > 64 bits.
> > > >
> > > > Putting it at 64 bits would make it rather unlikely that the ceiling
> > > > is ever hit. 1ms network latency * 2^64 is "sizable". Some machines
> > > > may have to do their own 64 bit arithmetic if they don't have 64
> > > > bit libraries (or 64 bit processors), but that's not a serious issue.
> > >
> > > Yes, but arguing against myself, I'm not sure I want to -require- 64
> > > bits, esp. if I'm trying to work with an existing group manager that
> > > has a smaller native width. GRITS/NATALIE is not supposed to be
> > > Linux specific, so I kind of don't like wiring things in like
> > > generation size.
> > >
> > > Any opinions?
> >
> > I think that on any platform double-word arihmetic is simple and it would be
> > best to avoid any potential wrap, although i think that testing should be done
> > with an 8 bit number to ensure that the wrap works.
> Fair enough. Put it as a typedef or #define in the initial
> implementation to at least test it, and then publish it switched to the
> 64 bit mode. 1 ms latency between machines (as a minimum time for
> reconfig duration) comes to 584 million years before we wrap. I know
> people in previous decades have hit y2k problems by considering that
> their code couldn't possibly be still in use for 20 or 30 years, but
> perhaps we're safe that our code will be long forgotten after 584
> million. :) Bit rot should take care of that nicely enough, if nothing
> else.
The point I'm trying to address is the pre-existing cluster manager that
uses a 16 bit generation counter, then wraps it itself. Do we -require-
it to change to 32 or 64 bits to use GRITS? That seems like a pretty
big demand, and we've been trying to be accomodating to existing systems.
This is irrelevant to the Linux guys, but important for other places.
-dB