Mailing List Archive

> 1) Change xcs to use unix domain sockets.

The original thought behind using ip sockets behind xcs was in
planning for cluster management. Obviously there are other ways to
map this in, I'm not terribly fussy on this point.

> 2) Add support to xcs to export ptys (storing info in the filesystem
> much the same way xenctld does)

I would much rather see xcs only handle control messages, and see the
console stuff broken off onto a seperate split driver. ptys/ip
sockets/whatever can then be mapped in at the backend independent of
how the control plane works. I just chatted with keir about this, and
he agrees that pulling console messages off onto their own rings and
providing a separate backend for them would be a good thing.

> 3) Change xenctld tools to use xcs.

sure... although i think there is a fair bit more involved in
create/destroy than what those tools are providing.

> 4) Factor out most of xen interaction in xcs to standard libraries.

Much of the xen interaction in xcs _is_ already in shared libraries
(libxc and libxutil). The control channels can only be safely bound
by a single consumer in dom0 -- xcs just serves to multiplex this
access. The interfaces in xcs could probably do with some cleaning
up, as they are reflective of my pulling a lot of the structure out of
the python code in December. I'm not sure what bits of it you'd like
to see generalized out of xcs though... can you be more specific?

> I see a three level architecture, the first level being highly portable
> libraries that simplify interacting with Xen. This would target every
> platform Xen runs on.
> ...

This is what we have been shooting for with the new control tools: 1.
libxc/xutil, 2. xcs, 3. higher-level tools.

> Thoughts? I'm willing to code these things up. Just want to make sure
> it's agreeable first.

Our current plan with the control tools is in fixing up a couple of
things (1) how vm state is represented in dom0, and (2) how easy it is
to add and maintain new tools, drivers, etc.

xcs is a first step, in that it allows new tools be run along side xend.

The next step, coming soon, will be a 'persistent store' which will be
a repository for vm state. This will hold things like active domain
configurations, device details, etc.

In addition to this, we have been discussing the option of adding
endpoint addressing to the control messages. So driver setup, for
instance, would move toward a control tool pushing details into the
store prior to starting the domain. the frontend driver would then
query the store for the backend details, and then address the backend
directly. This should make extending the tools considerably easier.

This is all very high on the to do list, and should start to emerge
over the next while. It would be great to discuss design points in
more detail on the list.

Things are a little busy here right now, so i hope this isn't too brief. ;)

best,
a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Andrew Warfield wrote:

> > 1) Change xcs to use unix domain sockets.
>
> The original thought behind using ip sockets behind xcs was in
> planning for cluster management. Obviously there are other ways to
> map this in, I'm not terribly fussy on this point.

how will this work if we run a 256-node xen cluster (which we will be
doing here for testing)? Wouldn't we want the IP socket?

> I would much rather see xcs only handle control messages, and see the
> console stuff broken off onto a seperate split driver. ptys/ip
> sockets/whatever can then be mapped in at the backend independent of how
> the control plane works. I just chatted with keir about this, and he
> agrees that pulling console messages off onto their own rings and
> providing a separate backend for them would be a good thing.

make sense to me, again I sure hope we can keep things so that we can put
lightweight xend's on bproc slave nodes (i.e. in the end we'd like to be
python-free) and control from a single front-end node via tcp.

I think python-free is beyond the scope of this discussion, just thought
I'd mention it -- it's a real problem to run python tools in bproc
clusters.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 10:39, Andrew Warfield wrote:
> The original thought behind using ip sockets behind xcs was in
> planning for cluster management. Obviously there are other ways to
> map this in, I'm not terribly fussy on this point.

Yeah, I figured as much. My only reason for domain sockets is that in
the minimal configuration, I figured that domain-0 should not expose any
network sockets. I'm fine with exporting as both a domain and an ip
socket (based on a command line option).

> > 2) Add support to xcs to export ptys (storing info in the filesystem
> > much the same way xenctld does)
>
> I would much rather see xcs only handle control messages, and see the
> console stuff broken off onto a seperate split driver. ptys/ip
> sockets/whatever can then be mapped in at the backend independent of
> how the control plane works. I just chatted with keir about this, and
> he agrees that pulling console messages off onto their own rings and
> providing a separate backend for them would be a good thing.

Is this just what it sounds like?

> > 3) Change xenctld tools to use xcs.
>
> sure... although i think there is a fair bit more involved in
> create/destroy than what those tools are providing.

Oh yes. The vm-create tool does not work. I hacked it up last night
hoping that it would work for at least ramdisk-based kernels. I did not
get it to work.

> > 4) Factor out most of xen interaction in xcs to standard libraries.
>
> Much of the xen interaction in xcs _is_ already in shared libraries
> (libxc and libxutil). The control channels can only be safely bound
> by a single consumer in dom0 -- xcs just serves to multiplex this
> access. The interfaces in xcs could probably do with some cleaning
> up, as they are reflective of my pulling a lot of the structure out of
> the python code in December. I'm not sure what bits of it you'd like
> to see generalized out of xcs though... can you be more specific?

The thought was that xcs might be something that has to be rewritten.
If ptys are not in it than I guess it doesn't really matter.

> > I see a three level architecture, the first level being highly portable
> > libraries that simplify interacting with Xen. This would target every
> > platform Xen runs on.
> > ...
>
> This is what we have been shooting for with the new control tools: 1.
> libxc/xutil, 2. xcs, 3. higher-level tools.
>
> > Thoughts? I'm willing to code these things up. Just want to make sure
> > it's agreeable first.
>
> Our current plan with the control tools is in fixing up a couple of
> things (1) how vm state is represented in dom0, and (2) how easy it is
> to add and maintain new tools, drivers, etc.
>
> xcs is a first step, in that it allows new tools be run along side xend.

Great.

> The next step, coming soon, will be a 'persistent store' which will be
> a repository for vm state. This will hold things like active domain
> configurations, device details, etc.

What are the thoughts on how this will be exposed? Is this targeted for
user space or kernel space?

My thoughts along these lines was to use a directory and simulate a
/proc-like structure in user space. This has problems though.

> In addition to this, we have been discussing the option of adding
> endpoint addressing to the control messages. So driver setup, for
> instance, would move toward a control tool pushing details into the
> store prior to starting the domain. the frontend driver would then
> query the store for the backend details, and then address the backend
> directly. This should make extending the tools considerably easier.

Indeed.

> This is all very high on the to do list, and should start to emerge
> over the next while. It would be great to discuss design points in
> more detail on the list.
>
> Things are a little busy here right now, so i hope this isn't too brief. ;)

If there's things we can do to help out, let us know. I'm going to
working on xcs. If you've got feature requests or anything just say
them on the list, I'm sure someone will pick it up :-)

Thanks,

> best,
> a.
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 11:19, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Andrew Warfield wrote:
>
> > > 1) Change xcs to use unix domain sockets.
> >
> > The original thought behind using ip sockets behind xcs was in
> > planning for cluster management. Obviously there are other ways to
> > map this in, I'm not terribly fussy on this point.
>
> how will this work if we run a 256-node xen cluster (which we will be
> doing here for testing)? Wouldn't we want the IP socket?

It's not so much about whether there would be an IP interface at all,
more about what's the lowest possible interface. A daemon that uses the
domain socket interface could then export an IP interface.

Just running a single Xen server though, you probably don't want a port
open if it doesn't have to be.

Regards,
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Anthony Liguori wrote:

> It's not so much about whether there would be an IP interface at all,
> more about what's the lowest possible interface. A daemon that uses the
> domain socket interface could then export an IP interface.

another daemon would be a concern. On these slave nodes, the fewer daemons
the better.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Anthony Liguori wrote:

> Out of curiosity, why is that? Is it the storage overhead? Increasing
> the number of processes? Or simply just the additional administrative
> headache?

more daemons means more things to get run when you least want them to run.
It's just a good idea to keep the daemons to a minimum. Requiring a daemon
to translate a unix domain socket to ip just strikes me as very
unnecessary.

But, I'm not writing it, so past a point I think you should ignore my
requests if you don't like them :-)

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 14:44, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Anthony Liguori wrote:
>
> > It's not so much about whether there would be an IP interface at all,
> > more about what's the lowest possible interface. A daemon that uses the
> > domain socket interface could then export an IP interface.
>
> another daemon would be a concern. On these slave nodes, the fewer daemons
> the better.

Out of curiosity, why is that? Is it the storage overhead? Increasing
the number of processes? Or simply just the additional administrative
headache?

> ron
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 14:54, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Anthony Liguori wrote:
>
> > Out of curiosity, why is that? Is it the storage overhead? Increasing
> > the number of processes? Or simply just the additional administrative
> > headache?
>
> more daemons means more things to get run when you least want them to run.
> It's just a good idea to keep the daemons to a minimum. Requiring a daemon
> to translate a unix domain socket to ip just strikes me as very
> unnecessary.

Sure, it's not quite that simple though. The unix domain interface
would be very very low-level. The messages would pretty much get passed
directly to the domain's ring queue.

The ip interface might support SNMP or some other standardized network
protocol. In the very least, the IP interface would have to be
endian-neutral which means that it's going to be a more complicated
protocol.

> But, I'm not writing it, so past a point I think you should ignore my
> requests if you don't like them :-)

Feedback is always a good thing. I'm very curious to see what people
would want the control software to do.

Thanks,

> ron
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 13:54 -0700, Ronald G. Minnich wrote:
> more daemons means more things to get run when you least want them to run.
> It's just a good idea to keep the daemons to a minimum.

I prefer to instead keep the _function_ of daemons to a minimum, and run
as many purpose-built, lightweight daemons as needed to build the
functionality I want.

> Requiring a daemon to translate a unix domain socket to ip just
> strikes me as very unnecessary.

Reasonable people can differ, as it strikes me as perfectly reasonable
to build TCP translation as a separate layer on top of base UDP
communications. This let's those who want to avoid TCP from being
forced to deal with it, and makes any TCP policy encoded into the daemon
plugable via alternative implementations.

--
Jared Rhine <jared@wordzoo.com>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Jared Rhine wrote:

> I prefer to instead keep the _function_ of daemons to a minimum, and run
> as many purpose-built, lightweight daemons as needed to build the
> functionality I want.

I understand the preference from an esthetics point of view. From a
quantitative performance point of view, in an environment where you want
to minimize the activity that is running on a lightweight cluster node, it
has been shown that 1 is better than 2.

So, all other things being equal, if we can have 1 daemon with TCP sockets
instead of a daemon with unix domain sockets and a daemon that turns that
into TCP sockets, on a reasonably large cluster (1024 or more nodes)
you're much better off with 1 daemon. 1024 node clusters are not uncommon
these days.

Again, this is not an issue of esthetics, it's an issue of measured
performance.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 09:33, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Jared Rhine wrote:

> Again, this is not an issue of esthetics, it's an issue of measured
> performance.

Where's the performance issue? For instance, if you had one daemon that
forked off two children, would that be better than having two individual
processes? Is it simply a matter of having that extra file on disk?

I'm not sure I understand how this could create a performance problem.
I simply don't know that much about cluster-performance and am very
curious :-)

It seems like the only thing that would make sense is that it's multiple
processes but unfortunately this is the exact opposite performance
behavior of a n-way system...

> ron
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > Again, this is not an issue of esthetics, it's an issue of measured
> > performance.
>
> Where's the performance issue?

I think Ron was suggesting that the dual handoff of the data between the
UDP listener and the TCP listener would be foolish and avoidable and
inarguably is a measurable throughput degredation over a direct TCP
processing loop.

Which is a fair argument, though not raised in Ron's earlier objections.

It's certainly not unreasonable to have a direct TCP listener as well as
a UDP listener in the same daemon. Though single-purpose daemons are
better for maintainability and robustness, the minor extra code to have
both Unix domain and TCP branches is straightforward.

I do think that the "no TCP is my lightweight dom0" people should be
supported, though I'm not personally one of them. So a Unix domain
listener seems like it should be available through xcs. Making those
people build and maintainer and _alternative_ multiplexing switch isn't
good for anyone.

I now favor xcs supporting both listening approaches, though I'm not
volunteering code. I'd rather let the experts sort out the right
layering and then contribute on the tools that use the final mechanism.

> I'm not sure I understand how this could create a performance problem.
> I simply don't know that much about cluster-performance and am very
> curious :-)

Ron, is your performance concern because large clusters need to pass a
very high volume of control messages? For any low-volume situation,
this unix domain/TCP argument is a non-issue, mostly?

--
Jared Rhine <jared@wordzoo.com>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

[.Ronald == rminnich@lanl.gov on Mon, 24 Jan 2005 10:04:29 -0700 (MST)]

Jared> Ron, is your performance concern because large clusters need
Jared> to pass a very high volume of control messages?

Ronald> re-reading this, however, I think my concerns about the
Ronald> extra daemon shrink into insignificance when I think about
Ronald> Python-based Xend ...

I think most people are agreed that a non-Python control
infrastructure will be developed in the not-so-distant future. So
to the extent your concerns about multiple daemons apply in a
non-Python environment, they should be addressed.

Ronald> The issue is that, in any large scale cluster, if you have
Ronald> enough processes running besides your application on the
Ronald> nodes, then the act of scheduling those processes -- even
Ronald> for select calls that return instantly
Ronald> -- can derange the application performance.

It may be that clustering people need a whole different toolset. If
you were trying to minimize process count, then you probably wouldn't
even want xcs, a multiplexing switch anyway. Indeed, why multiplex at
all? You might use a single control mechanism, preferably combining
whatever subset of xcs, xensv, xm, etc you needed.

I'll leave it others to figure out what all this means for xcs's
future.

Thanks for your cluster scheduling overview; it was helpful.

-- jared@wordzoo.com

http://www.geekcode.com/geek.html
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/B
d s:++>+ a-
C++(++++)$ ULBSH++++ P+++ L+++ E++(+++) W++++ N- !o !K w !O M V
PS+++(-) PE++(--) Y+ PGP>++
t@ 5 X+ R>+ tv++>-- b>++ DI+ D- G
e++ h- r++>+++ y+++
------END GEEK CODE BLOCK------


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 10:55, Jared Rhine wrote:
> > > Again, this is not an issue of esthetics, it's an issue of measured
> > > performance.
> >
> > Where's the performance issue?
>
> I think Ron was suggesting that the dual handoff of the data between the
> UDP listener and the TCP listener would be foolish and avoidable and
> inarguably is a measurable throughput degredation over a direct TCP
> processing loop.

You're not going to have a direct listener though on TCP. I think you
mean unix domain sockets too instead of UDP. The current message
passing architecture assumes the messages are originating from the
native architecture.

They do not take into account things like endianness into
consideration. They certainly don't security into account at all. To
export a proper TCP interface requires a higher level protocol. The TCP
interface will have to do a certain amount of work to take these things
into account.

I think we all agree that we need a TCP interface.

The only real question is whether you export a lower level protocol
interface via something like domain sockets. Domain sockets are
extremely fast on most platforms (usually reducing to something like a
memcpy). Considering the message size here I do not think you can make
a performance argument for having a TCP daemon go through a domain
socket.

If we agree that we need a higher level protocol for TCP, then it
becomes pretty apparently that there's going to be desire for multiple
protocols. Maybe a less security conscious protocol for isolated
clusters (like xcs implements right now) verses a more hardened protocol
for enterprise usage.

Using multiple daemons here becomes a no-brainer. You just start the
appropriate daemon for whatever protocol you wish to export. Properly
architected, multi daemons could even be run simultaneously to support
multiple sets of control tools.

> Ron, is your performance concern because large clusters need to pass a
> very high volume of control messages? For any low-volume situation,
> this unix domain/TCP argument is a non-issue, mostly?

I have a feeling that the concern is that there's a general fear that
more daemons == more resource usage == more administrative overhead. I
don't think this is true though. I think it simply depends on how
things are architected.

--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 11:52, Jared Rhine wrote:
> Ronald> The issue is that, in any large scale cluster, if you have
> Ronald> enough processes running besides your application on the
> Ronald> nodes, then the act of scheduling those processes -- even
> Ronald> for select calls that return instantly
> Ronald> -- can derange the application performance.

This is why schedulers have priority. Just bump up the priority of your
desired app. Also, with the Linux O(1) scheduler, a process blocked on
I/O (at least, by my understanding) should have no impact on other
processes scheduling.

--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

As I see it, we have two (or more) camps/uses for xen. One (micro) that
is dealing primarily with a relatively small number of machines -
typically one and is concerned with speed and security of the local
machine. The other (macro) is HPC or large enterprise/server farm type
setups that requires out of the physical box remote connectivity. The
third camp just wants something reliable and easy to use (probably -
though they have been quiet up to this point so I'm just guessing
here). The real question is can one tool be made to fit all, or is the
gap so large that two different tools are required.

B.



On Mon, 2005-01-24 at 15:07, Anthony Liguori wrote:
> On Mon, 2005-01-24 at 10:55, Jared Rhine wrote:
> > > > Again, this is not an issue of esthetics, it's an issue of measured
> > > > performance.
> > >
> > > Where's the performance issue?
> >
> > I think Ron was suggesting that the dual handoff of the data between the
> > UDP listener and the TCP listener would be foolish and avoidable and
> > inarguably is a measurable throughput degredation over a direct TCP
> > processing loop.
>
> You're not going to have a direct listener though on TCP. I think you
> mean unix domain sockets too instead of UDP. The current message
> passing architecture assumes the messages are originating from the
> native architecture.
>
> They do not take into account things like endianness into
> consideration. They certainly don't security into account at all. To
> export a proper TCP interface requires a higher level protocol. The TCP
> interface will have to do a certain amount of work to take these things
> into account.
>
> I think we all agree that we need a TCP interface.
>
> The only real question is whether you export a lower level protocol
> interface via something like domain sockets. Domain sockets are
> extremely fast on most platforms (usually reducing to something like a
> memcpy). Considering the message size here I do not think you can make
> a performance argument for having a TCP daemon go through a domain
> socket.
>
> If we agree that we need a higher level protocol for TCP, then it
> becomes pretty apparently that there's going to be desire for multiple
> protocols. Maybe a less security conscious protocol for isolated
> clusters (like xcs implements right now) verses a more hardened protocol
> for enterprise usage.
>
> Using multiple daemons here becomes a no-brainer. You just start the
> appropriate daemon for whatever protocol you wish to export. Properly
> architected, multi daemons could even be run simultaneously to support
> multiple sets of control tools.
>
> > Ron, is your performance concern because large clusters need to pass a
> > very high volume of control messages? For any low-volume situation,
> > this unix domain/TCP argument is a non-issue, mostly?
>
> I have a feeling that the concern is that there's a general fear that
> more daemons == more resource usage == more administrative overhead. I
> don't think this is true though. I think it simply depends on how
> things are architected.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 16:39 +0000, Andrew Warfield wrote:

Thanks for the info.


> > 4) Factor out most of xen interaction in xcs to standard libraries.
>
> Much of the xen interaction in xcs _is_ already in shared libraries
> (libxc and libxutil). The control channels can only be safely bound
> by a single consumer in dom0 -- xcs just serves to multiplex this
> access. The interfaces in xcs could probably do with some cleaning
> up, as they are reflective of my pulling a lot of the structure out of
> the python code in December. I'm not sure what bits of it you'd like
> to see generalized out of xcs though... can you be more specific?
>
> > I see a three level architecture, the first level being highly portable
> > libraries that simplify interacting with Xen. This would target every
> > platform Xen runs on.
> > ...
>
> This is what we have been shooting for with the new control tools: 1.
> libxc/xutil, 2. xcs, 3. higher-level tools.

Is it reasonable to expect that a port to another machine architecture,
assuming the libxc/xutil are provided, then xcs and higher-level tools
should just follow?

>
> > Thoughts? I'm willing to code these things up. Just want to make sure
> > it's agreeable first.
>
> Our current plan with the control tools is in fixing up a couple of
> things (1) how vm state is represented in dom0, and (2) how easy it is
> to add and maintain new tools, drivers, etc.
>
> xcs is a first step, in that it allows new tools be run along side xend.
>
> The next step, coming soon, will be a 'persistent store' which will be
> a repository for vm state. This will hold things like active domain
> configurations, device details, etc.

What form do you see this persistent store taking? Is it just for
currently present VM's, or can it be used also to pre-configure
VM's and/or have pre-defined VM's? I assume some sort of filesystem
backed entity is to be used for the persistent store, correct?
Format, content, etc.?
>
> In addition to this, we have been discussing the option of adding
> endpoint addressing to the control messages. So driver setup, for
> instance, would move toward a control tool pushing details into the
> store prior to starting the domain. the frontend driver would then
> query the store for the backend details, and then address the backend
> directly. This should make extending the tools considerably easier.
>
> This is all very high on the to do list, and should start to emerge
> over the next while. It would be great to discuss design points in
> more detail on the list.

Yes. This is a good overview. More details especially in regards to
the persistent store, and the set of tools being built would add to
this discussion. Also, it is likely that I (and others) can provide
development and testing assistance if we understand what is being
developed and where our contributions can be used.

Thanks.

Michael
>
> Things are a little busy here right now, so i hope this isn't too brief. ;)
>
> best,
> a.

--
Michael Hohnbaum 503 578 5486
hohnbaum@us.ibm.com t/l 775 5486



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Anthony Liguori wrote:
> They do not take into account things like endianness into
> consideration. They certainly don't security into account at all. To
> export a proper TCP interface requires a higher level protocol. The TCP
> interface will have to do a certain amount of work to take these things
> into account.
>
> I think we all agree that we need a TCP interface.

No. In my implementation I think I have shown that you can run your
cluster with a much more minimal network-interface.

My current implementation is less than 100 lines of network-facing code
(using ICMP payloads, but the same could be achieved with UDP), though
that increases to about 300 if you count in the SHA-1 implementation
that I am using for verifying customer credentials. So 300 lines is for
a complete solution including crypto and accounting/payment.

My nodes answer ARPs and pings, but that is all.

# nmap -sS blackbox07

Starting nmap 3.55 ( http://www.insecure.org/nmap/ ) at 2005-01-25 18:26 PST
All 1660 scanned ports on blackbox07 are: filtered

Nmap run completed -- 1 IP address (1 host up) scanned in 71.969 seconds

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Jacob Gorm Hansen wrote:

> My current implementation is less than 100 lines of network-facing
> code (using ICMP payloads, but the same could be achieved with UDP),
> though that increases to about 300 if you count in the SHA-1
> implementation that I am using for verifying customer credentials. So
> 300 lines is for a complete solution including crypto and
> accounting/payment.

That's certainly interesting. Are you using ICMP simply for high-level
management or are you passing event channel messages via ICMP? Are you
just punting issues of reliability?

I agree with you, TCP is not the only network solution. However, it
seems logically that a large number of people will want a TCP solution :-)

Regards,

--

Anthony Liguori
anthony@codemonkey.ws



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Anthony Liguori wrote:
> Jacob Gorm Hansen wrote:
>
>> My current implementation is less than 100 lines of network-facing
>> code (using ICMP payloads, but the same could be achieved with UDP),
>> though that increases to about 300 if you count in the SHA-1
>> implementation that I am using for verifying customer credentials. So
>> 300 lines is for a complete solution including crypto and
>> accounting/payment.
>
>
> That's certainly interesting. Are you using ICMP simply for high-level
> management or are you passing event channel messages via ICMP? Are you
> just punting issues of reliability?
>
> I agree with you, TCP is not the only network solution. However, it
> seems logically that a large number of people will want a TCP solution :-)
>

I am using TCP, but I am doing both bootstrap+shutdown of new domains,
migration, and accounting mostly within domUs. I send an ICMP payload
with an initial ticket/capability describing the max memory, disk
extents, external IP etc., and the ICMP server in dom0 decodes/verifies
that, an fires up a domU with a minimal TPC/IP stack and server inside.
The domain builder is then uploaded on that connection, takes over, and
handles the bootstrap from there. Same is true for migration.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Hi Michael,

> Is it reasonable to expect that a port to another machine architecture,
> assuming the libxc/xutil are provided, then xcs and higher-level tools
> should just follow?

This seems pretty reasonable to me. Most of what happens in xend
right now is managing the state of the vms, much of the real mechanism
(domain building, suspension, migration, etc.) is really inside the
underlying libraries (xc, xutil). The only vaguely dependent thing i
can think of that isn't masked by these libraries is the shared memory
interface ot the control rings accessed through xcs. This isn't a
very big deal though.

> What form do you see this persistent store taking? Is it just for
> currently present VM's, or can it be used also to pre-configure
> VM's and/or have pre-defined VM's? I assume some sort of filesystem
> backed entity is to be used for the persistent store, correct?
> Format, content, etc.?

We have had some discussion about this, and Mike Wray has written some
design documentation on the new store plan. I think the principle
goal of the store is to encapsulate all of the state relating to the
currently running set of VMs in a single place, that has clear API for
access. Our basic view of this at the moment is a hierarchical set of
key-value pairs that looks remarkably like a file system. ;)

Having the store worked in properly should 1. make rebooting dom0 with
active VMs unaffected more possible, 2. make adding new functionality
(e.g. new split device types) more straightforward, and 3. make
writing tools that need state information, and 4. scaling control up
to a cluster all a lot more tractable.

Including more long-lived stuff like config files seems like a
potentially reasonable thing to do as well.

> Yes. This is a good overview. More details especially in regards to
> the persistent store, and the set of tools being built would add to
> this discussion. Also, it is likely that I (and others) can provide
> development and testing assistance if we understand what is being
> developed and where our contributions can be used.

Fantastic! We would be happy to have the help. We are still in the
process of hashing out exactly what the new interfaces should look
like and how to go about moving to a new set of tools, potentially for
the 3.0 release. Probably the best way forward is for us to post
design sketches to the list as they become available, and iteratively
move towards bits of work that people can bite off. Any specific
ideas that you guys want to throw in now would be more than welcome as
well.

sound okay?

a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> In reality, what's the different between a domain that's suspended to
> disk and a clone domain that just hasn't been started yet.
>
> Does the current persistent store architecture attempt to solve this
> problem? Does it attempt to solve both problems?

No. There has been no discussion of storing suspended domains. For
the moment, i think it's probably best to design the store to contain
runtime state for the currently executing set of VMs on the physical
host. So records in the store are persistent across daemon restarts,
and dom0 restarts.

I don't think any of use really imagine the store as being a very
complicated bit of code at all. The bit of design that needs a little
more thought is how things like device driver setup should look with
the new control tools. This isn't necessarily terrible -- but
changing e.g. device channel negotiation at driver setup is going to
be a lot less work this round (with three split drivers) than it is
going to be if we need yet another set of control tools for Xen 4. ;)

a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 08:33, Andrew Warfield wrote:
> Having the store worked in properly should 1. make rebooting dom0 with
> active VMs unaffected more possible, 2. make adding new functionality
> (e.g. new split device types) more straightforward, and 3. make
> writing tools that need state information, and 4. scaling control up
> to a cluster all a lot more tractable.

It seems like there's two distinct problems.

1) Managing domain specific state for running domains.

This can be done easily enough since a domid is a unique identifier.
Any associative database would work just fine.

2) Managing domain specific state for non-running domains.

Since non-running domains have to have some sort of disk representation
(either a config file of some sort or a snapshot image) this also is
pretty easy.


I'm not convinced there's a great way to merge the two though. It gets
really hairy once you start doing things like cloning or migration.
There's a lot of subtle things like reassigning MAC addresses that gets
nasty.

In reality, what's the different between a domain that's suspended to
disk and a clone domain that just hasn't been started yet.

Does the current persistent store architecture attempt to solve this
problem? Does it attempt to solve both problems?

I'd like to see a persistent store that just addressed problem #1. I
think once you get into trying to solve #2, you'll create a solution
that won't work for everyone.

I'd like to see Xen with a tool that can create domains from a
configuration file but at the same time, it would be nice if the
persistent store worked well with domain's that are created by hand.

Regards,

--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 13:11, Andrew Warfield wrote:
> > In reality, what's the different between a domain that's suspended to
> > disk and a clone domain that just hasn't been started yet.
> >
> > Does the current persistent store architecture attempt to solve this
> > problem? Does it attempt to solve both problems?
>
> No. There has been no discussion of storing suspended domains. For
> the moment, i think it's probably best to design the store to contain
> runtime state for the currently executing set of VMs on the physical
> host. So records in the store are persistent across daemon restarts,
> and dom0 restarts.

Excellent! Then I have no worries at all about it.

You mentioned the current design looks a lot like a file system, any
thoughts on actually using the file system? Either within the normal
file system (within something like /var/xen/<domid>) or as a pid-like
/proc entry (like /proc/xen/<domid>).

Regards,
--
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel