Hi,
On Mon, 31 Jan 2000 14:37:24 -0500, Keith Barrett <kbarrett@redhat.com>
said:
> That was my thought. I think this is a good discussion topic, and I
> know I need help in either understanding why we won't use messaging
> (because we may also get asked by outsiders), or how clustering may
> future impact a personal effort on my part to provide a Message
> Queuing environment for Linux. I honestly don't understand why we
> would not want to layer ourselves from the commuications system. I
> also got the impression that one or two people in our Clustering
> symposium agreed.
> I will, of course, live with whatever Clustering architecture is
> designed.
>> Umm, sockets *are* the Unix transport-independent messaging layer.
> Forgive my ignorance here (I honestly don't know), but do socket calls
> support native ATM?
Yes, they do.
> I know they don't support direct ethernet without IP
Yes, they do, but you need to have root privileges to use SOCK_RAW (as
there are serious security implications in giving normal users raw
access to the wire!).
> (which might have offered some performance opportunities). Do future
> protocols always get included in the socket APIs?
The socket API is generic. You can define any protocol you want by
specifying a new address family and new protocol family constant on the
local system.
> I know from personal experience that socket calls do not work
> universally the same when you cross multiple platforms. For example;
> Some platforms (Motorola or AS400 if I remember correctly) did not let
> you create a pipe larger than 1k bytes; forcing you to deal with your
> own frame disection and reassembly (most other platforms allow a pipe
> at least 32k long).
I'm sold on the advantages. However, IP sockets _are_ a lowest common
denominator. The question that bothers me is, do we want to provide
messaging eventually as an API for clustered applications, or do we want
to provide it now and build the whole cluster infrastructure on top of
it?
The problem is that if we go the latter route, then we can code and test
_nothing_ until the message queues are in place. Using message queues
as infrastructure has its own set of problems, too --- cluster service
daemons which have to participate in recovery necessarily have to have
very different comms behaviour over a cluster transition than cluster
applications running above that level. A lock manager will need to
reset all of its cluster interconnections when recovery occurs. An
application using message queuing, on the other hand, should simply not
notice a cluster transition if it doesn't involve the loss of any node
it is communicating with.
> Some platforms block listeners, others support multiple listen
> connects. Also; not all systems support sockets (some use
> streams).
Streams-based systems almost always provide a socket API in addition.
> There's also an ad-hoc way of using messaging and a proxy Linux
> system to immediately bring incompatible systems (like WinNT) into
> the cluster until someone codes native services for it. This has
> many political advantages.
Windows supports sockets.
> But on their own, these may not be enough of a case for a messaging
> subsystem in clustering (at least, not initially. It does become more
> important when you want non-linux (and especially non-unix) systems
> to join clustering. This is the reason I brought this up. Even DEC,
> when it was looking into non-VMS cluster memberships, was considering
> messaging. If we will be needing a messaging layer eventually,
> perhaps its API should be designed and coded against now to save
> rework?
Right --- it's getting the layering right which matters at the moment.
So, here are some of the questions I think we need to settle before we
can make that judgement:
How much work is involved in implementing enough MQ to get the cluster
services working? How much performance will be lost in its
implementation? How easily can we settle the API? Given that the
reality is that all our expected initial adopters of clustering will be
running exclusively IP, how transparent can we make the message queuing
to them? (We do NOT want to expose a new addressing scheme to IP
users!)
> Do we really trust that sockets are a good universal communuications
> layer?
Yes. Furthermore, in the Unix world, socket support is simply required:
we _have_ to support it so that existing applications can use clustering
easily.
> Do we trust that all future protocols will be sockets enabled?
Yes, although that does not mean that sockets will always be the most
efficient way of accessing future protocols. (But the same is true of
MQ: if you need every last bit of performance over a VIA interconnect,
you'll talk raw VIA in either case).
The question is not whether or not sockets are a good universal layer.
They _are_ pretty much universal, but they are also very much a lowest
common denominator.
The question which matters _right now_ is really how much infrastructure
we build on top of sockets for our initial core cluster services. That
initial core will necessarily be built on top of sockets, but we can add
as much abstraction there as we want in order to make things more
portable for the future.
My own feeling is that the abstraction we need to provide has to include
clean large message support and transparent connection opening. The
semantics _should_ be that we can pick a cluster member and send any
message, treating the messaging layer as a reliable ordered datagram
service but using tcp underneath. A cluster transition should be
visible just as a barrier reset in that layer: connection management
should be abstracted.
That is what is _required_ for the cluster services v1. What we eventually
want to offer to applications in our full cluster infrastructure
services may be very different.
--Stephen
On Mon, 31 Jan 2000 14:37:24 -0500, Keith Barrett <kbarrett@redhat.com>
said:
> That was my thought. I think this is a good discussion topic, and I
> know I need help in either understanding why we won't use messaging
> (because we may also get asked by outsiders), or how clustering may
> future impact a personal effort on my part to provide a Message
> Queuing environment for Linux. I honestly don't understand why we
> would not want to layer ourselves from the commuications system. I
> also got the impression that one or two people in our Clustering
> symposium agreed.
> I will, of course, live with whatever Clustering architecture is
> designed.
>> Umm, sockets *are* the Unix transport-independent messaging layer.
> Forgive my ignorance here (I honestly don't know), but do socket calls
> support native ATM?
Yes, they do.
> I know they don't support direct ethernet without IP
Yes, they do, but you need to have root privileges to use SOCK_RAW (as
there are serious security implications in giving normal users raw
access to the wire!).
> (which might have offered some performance opportunities). Do future
> protocols always get included in the socket APIs?
The socket API is generic. You can define any protocol you want by
specifying a new address family and new protocol family constant on the
local system.
> I know from personal experience that socket calls do not work
> universally the same when you cross multiple platforms. For example;
> Some platforms (Motorola or AS400 if I remember correctly) did not let
> you create a pipe larger than 1k bytes; forcing you to deal with your
> own frame disection and reassembly (most other platforms allow a pipe
> at least 32k long).
I'm sold on the advantages. However, IP sockets _are_ a lowest common
denominator. The question that bothers me is, do we want to provide
messaging eventually as an API for clustered applications, or do we want
to provide it now and build the whole cluster infrastructure on top of
it?
The problem is that if we go the latter route, then we can code and test
_nothing_ until the message queues are in place. Using message queues
as infrastructure has its own set of problems, too --- cluster service
daemons which have to participate in recovery necessarily have to have
very different comms behaviour over a cluster transition than cluster
applications running above that level. A lock manager will need to
reset all of its cluster interconnections when recovery occurs. An
application using message queuing, on the other hand, should simply not
notice a cluster transition if it doesn't involve the loss of any node
it is communicating with.
> Some platforms block listeners, others support multiple listen
> connects. Also; not all systems support sockets (some use
> streams).
Streams-based systems almost always provide a socket API in addition.
> There's also an ad-hoc way of using messaging and a proxy Linux
> system to immediately bring incompatible systems (like WinNT) into
> the cluster until someone codes native services for it. This has
> many political advantages.
Windows supports sockets.
> But on their own, these may not be enough of a case for a messaging
> subsystem in clustering (at least, not initially. It does become more
> important when you want non-linux (and especially non-unix) systems
> to join clustering. This is the reason I brought this up. Even DEC,
> when it was looking into non-VMS cluster memberships, was considering
> messaging. If we will be needing a messaging layer eventually,
> perhaps its API should be designed and coded against now to save
> rework?
Right --- it's getting the layering right which matters at the moment.
So, here are some of the questions I think we need to settle before we
can make that judgement:
How much work is involved in implementing enough MQ to get the cluster
services working? How much performance will be lost in its
implementation? How easily can we settle the API? Given that the
reality is that all our expected initial adopters of clustering will be
running exclusively IP, how transparent can we make the message queuing
to them? (We do NOT want to expose a new addressing scheme to IP
users!)
> Do we really trust that sockets are a good universal communuications
> layer?
Yes. Furthermore, in the Unix world, socket support is simply required:
we _have_ to support it so that existing applications can use clustering
easily.
> Do we trust that all future protocols will be sockets enabled?
Yes, although that does not mean that sockets will always be the most
efficient way of accessing future protocols. (But the same is true of
MQ: if you need every last bit of performance over a VIA interconnect,
you'll talk raw VIA in either case).
The question is not whether or not sockets are a good universal layer.
They _are_ pretty much universal, but they are also very much a lowest
common denominator.
The question which matters _right now_ is really how much infrastructure
we build on top of sockets for our initial core cluster services. That
initial core will necessarily be built on top of sockets, but we can add
as much abstraction there as we want in order to make things more
portable for the future.
My own feeling is that the abstraction we need to provide has to include
clean large message support and transparent connection opening. The
semantics _should_ be that we can pick a cluster member and send any
message, treating the messaging layer as a reliable ordered datagram
service but using tcp underneath. A cluster transition should be
visible just as a barrier reset in that layer: connection management
should be abstracted.
That is what is _required_ for the cluster services v1. What we eventually
want to offer to applications in our full cluster infrastructure
services may be very different.
--Stephen