Mailing List Archive

On Tue, 25 Jan 2005, Jacob Gorm Hansen wrote:

> with Xen increasingly depending on Linux for bootstrap, drivers, packet
> filtering etc., would it make sense to have the option of compiling Xen as a
> Linux kernel module, like in VMWare or coLinux?

that's weird, I just got this message a second time, but I still don't
think it is a good idea :-)

ah, sourceforge.

Anyway, I think the hypervisor should stand alone. Just my feeling about
it.


ron



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

>with Xen increasingly depending on Linux for bootstrap, drivers, packet
>filtering etc., would it make sense to have the option of compiling Xen
>as a Linux kernel module, like in VMWare or coLinux?

Huh? Which kind of vmware? Afaik the hosted (type II) versions of
vmware when running on a linux host have some modules which get installed
but the VMM itself is not a module. And coLinux is basically a windows
device driver which does task switching - a very clever and useful piece
of software but not really a Linux kernel module.

Perhaps you mean: would it make sense to have a type II version of
Xen? I certainly see no particular reason to do this, but no particular
reason not to either. I'm not sure how much code similarity there
would be though...

>It seems this would give similar performance to Xen 1.2, while retaining
>most of the benefits of the NGIO model (i.e. not having to port
>drivers).

Maybe - I guess it depends on what you mean. If you have:

[ VM1 ] [ VM2 ] .... [ VMN ]

[ new type II version of Xen ]

[ linux kernel ]

[ hardware ]

then you require a way for VMx to communicate the new Xen thing,
which then needs to syscall into the linux kernel. I'm not sure
what VMx<->Xen comms would look like, or how it would perform. If
you retain safety it seems like you might end up with the performance
of UML, which if you go for 'high performance' then you may need to
turn off the safety catch.

How did you see this working?

What aspects of performance under Xen are you finding unacceptable?

There will always be an overhead involved in running N operating
systems and apps on a machine compared with just 1. Indeed, if
you really want 'blistering performance' you may find that even
the overhead of a general purpose OS is too much. Application-specific
OSes can increase performance (as can dynamic application-specific code
path optimization, see e.g. synthesis, wiggin-redstone, etc).

>The only downside would be the lack of driver isolation, but
>most people would be willing to live with that is my guess (plus as long
>as there is no IO-MMU a bad driver is still able to take down the
>complete system anyhow).

Well isolation (both security and performance) are two explicit
design goals of Xen. If you want to have the illusion of multiple
kernels without these properties, you can use linux vservers or
BSD jail.

We're quite keen to evolve Xen in a way which makes it easier to
run multiple configurations, but mainly in the space that /increases/
isolation (e.g. driver domains) rather than the other way.

>I imagine this could be done in a way that would also work under other
>host-OSes, like *BSD or Windows.

Again, I'm not sure how much code-base similarity there would be with
either current Xen or the type-II variant that you propose above.

One nice thing about a type-I (unhosted) hypervisor is that you
are - in principle at least - independent of the OSes you host.
This means one can have a dom0 running Linux, BSD, Plan9 or even
Windows. With type-II hypervisors you effectively need a new
hypervisor for every hosted OS type (e.g. VMware workstation).


>Any comments?

It might be useful if you were to state what precisely the problem
is that you wish to solve, why existing solutions are insufficient,
and how your proposed solution would solve the problem. I'm not sure
I really understand the answers to any of these at the moment.


cheers,

S.



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Secondly, there have been repeated reports on this list of
> people having
> problems with lower performance in domU than in dom0, perhaps due to
> cheap hardware, perhaps just due to misconfiguration,

I think most of these problems were down to debugging accidently getting
enabled in the stable build. We seem to be back to normal performance
now.

I think there are still some issues with particular ioapic's, but this
code is about to get rewritten and moved into dom0 anyhow.

> and the
> figures on
> the Xen website have not been updated to reflect what the actual
> situation is, so I guess nobody knows what the overhead will
> look like
> for a specific type of application.

The performance figures for SPECCPU, OSDB/PostgreSQl, SPECWeb99/Apache,
Postmark etc are pretty much identical between 2.0 and 1.2 --- see the
"Safe Hardware Access with the Xen Virtual Machine Monitor" paper. The
new IO model tends to burn more CPU for a given IO rate, but under high
load things pipeline reasonably nicely and the privilege transitions are
amortized. Latency does suffer, but it was never that great under 1.2
anyhow. Smarter NICs will help this a lot (e.g. see the Arsenic paper by
myself and Keir).

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Steven Hand wrote:

> Maybe - I guess it depends on what you mean. If you have:
>
> [ VM1 ] [ VM2 ] .... [ VMN ]
>
> [ new type II version of Xen ]
>
> [ linux kernel ]
>
> [ hardware ]

Sorry if this came out sounding as a bit of a troll, anyway, my
suggested setup would look like this:

[ VM1 ] [ VM2 ] .... [ VMN ]

[ Xen + linux kernel ]

[ hardware ]


> then you require a way for VMx to communicate the new Xen thing,
> which then needs to syscall into the linux kernel. I'm not sure
> what VMx<->Xen comms would look like, or how it would perform. If
> you retain safety it seems like you might end up with the performance
> of UML, which if you go for 'high performance' then you may need to
> turn off the safety catch.

Right now Xen is mapped somewhere in top of memory, I am not sure how
domains are kept out of there, but I suppose it has to do with segments.
The good thing about that is that hypercalls are cheap, and in Xen1.x
I/O was cheap as well.

My suggestion/question was a) why don't we just put a full Linux up
there, including drivers, and b) can we provide the Xen hypercall
interface on top of other OSes as well?

> How did you see this working?

For Linux, I would relocate it to the top X megs of memory, and I would
merge the Xen and Linux syscall handlers, essentially supporting two
process models under the same OS. I would not map all of memory to
Linux, just the pages it needs for its own stuff. For a driver OS, this
would be fine, if you want to run applications as well there would be a
tradeoff between how much you map to Linux and how much to Xen domains.

> What aspects of performance under Xen are you finding unacceptable?

I generally find performance acceptable, but as I said there are cases
where there appears to be some friction against the goals of Xen (driver
isolation) and the goals of the application (throughput, low latency).

> Well isolation (both security and performance) are two explicit
> design goals of Xen. If you want to have the illusion of multiple
> kernels without these properties, you can use linux vservers or
> BSD jail.

I would argue that you could get the same level of isolation (except
from driver isolation) if you merge the two, while achieving the same IO
performance as the monolithic model, and still be able to reuse existing
driver code.

>>I imagine this could be done in a way that would also work under other
>>host-OSes, like *BSD or Windows.
>
>
> Again, I'm not sure how much code-base similarity there would be with
> either current Xen or the type-II variant that you propose above.

It would still be interesting to reuse existing Xen guestOS ports on top
of different hypervisor implementations.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wednesday 26 January 2005 08:41, Steven Hand wrote:
> >with Xen increasingly depending on Linux for bootstrap, drivers, packet
> >filtering etc., would it make sense to have the option of compiling Xen
> >as a Linux kernel module, like in VMWare or coLinux?
>
> Huh? Which kind of vmware? Afaik the hosted (type II) versions of
> vmware when running on a linux host have some modules which get installed
> but the VMM itself is not a module. And coLinux is basically a windows
> device driver which does task switching - a very clever and useful piece
> of software but not really a Linux kernel module.

IIRC, the coLinux device driver is available both for Windows and as a Linux
kernel module. The same Linux kernel can then run in either environment.

> Maybe - I guess it depends on what you mean. If you have:
>
> [ VM1 ] [ VM2 ] .... [ VMN ]
>
> [ new type II version of Xen ]
>
> [ linux kernel ]
>
> [ hardware ]
>
> then you require a way for VMx to communicate the new Xen thing,
> which then needs to syscall into the linux kernel. I'm not sure
> what VMx<->Xen comms would look like, or how it would perform. If
> you retain safety it seems like you might end up with the performance
> of UML, which if you go for 'high performance' then you may need to
> turn off the safety catch.

Couldn't we have the "Xen module" hijack the interrupt-handling of the host
kernel (like the VMWare and coLinux modules themselves do AFAIK) in order to
handle hypercalls directly?

Device communications could be handled by backend / frontend drivers
essentially the same as the ones we use for vanilla Xen.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Thursday 27 January 2005 00:19, Jacob Gorm Hansen wrote:
> Sorry if this came out sounding as a bit of a troll, anyway, my
> suggested setup would look like this:
>
> [ VM1 ] [ VM2 ] .... [ VMN ]
>
> [ Xen + linux kernel ]
>
> [ hardware ]

I am interested in fast VMs. Assuming that you want dom0 to be '[ Xen + linux
kernel ]' I fail to see how your proposed architecture helps there.

> Right now Xen is mapped somewhere in top of memory, I am not sure how
> domains are kept out of there, but I suppose it has to do with segments.

As I understand this xen runs in ring0 and pushes the guests kernels one ring
up into ring1 and then uses traps to allow the guest OSes to trap into the
hypervisor as necessary.

> The good thing about that is that hypercalls are cheap, and in Xen1.x
> I/O was cheap as well.

Cheap where? In dom0 or the VMs?

> My suggestion/question was a) why don't we just put a full Linux up
> there, including drivers, and b) can we provide the Xen hypercall
> interface on top of other OSes as well?

a) Because that impacts security and robustness a lot. Security and
robustness are the two attributes I want in software, especially a kernel.
This is even more true for a hypervisor.

b) I don't understand what you are going at in b).

> > What aspects of performance under Xen are you finding unacceptable?
>
> I generally find performance acceptable, but as I said there are cases
> where there appears to be some friction against the goals of Xen (driver
> isolation) and the goals of the application (throughput, low latency).

Hmmm.... adding a layer of abstraction rarely improves throughput/latency. You
do it anyway to gain flexibility. I do not like you proposal as it sacrifices
flexibility I want for throughput/latency in a place I don't care about.

> > Well isolation (both security and performance) are two explicit
> > design goals of Xen. If you want to have the illusion of multiple
> > kernels without these properties, you can use linux vservers or
> > BSD jail.

Please keep those goals!

> I would argue that you could get the same level of isolation (except
> from driver isolation) if you merge the two, while achieving the same IO
> performance as the monolithic model, and still be able to reuse existing
> driver code.

I fail to see where monolithic kernel comes into this... I assume you are
referring to kernel running on a real machine instead of a virtual one.

Your proposal would force me to have all network traffic pass through dom0,
the system able to halt all VMs on the machine! I'd feel extremely nervous
with such a setup (OK, I am paranoid;-).

You do that to improve IO performance in dom0, which is the one virtual
machine that I do not need IO performance: dom0 is meant on my systems to be
able to setup VMs and nothing more (currently I use 16MiB of RAM for that
domain). All work is done in other domains. Those can no longer access
hardware directly with your proposal, thus it would even hurt IO performance
for me.

> It would still be interesting to reuse existing Xen guestOS ports on top
> of different hypervisor implementations.

The critical part of the hypervisor (from the perspective of a guest OS) is
the interface. That seems well defined and wouldn't need to change with your
proposal. So you should not need to modify domU OSes.

Your needs seem to be to have a fast dom0 and only spin of small sessions
occasionally, doing most of the work in dom0. Mine is to have several servers
sharing a piece of hardware. Dom0 is mostly idling while the other domains do
the heavy lifting. My hope is that I will see more proposals about reducing
the coupling of xen and dom0 (like being able to reboot dom0 without
effecting the other domains). I have no relation with the xen project apart
from using it, so I may only give my feelings.

--
Gruss,
Tobias

------------------------------------------------------------
Tobias Hunger The box said: 'Windows 95 or better'
tobias@aquazul.com So I installed Linux.
------------------------------------------------------------

> > Right now Xen is mapped somewhere in top of memory, I am not sure how
> > domains are kept out of there, but I suppose it has to do with segments.
>
> As I understand this xen runs in ring0 and pushes the guests kernels one
> ring up into ring1 and then uses traps to allow the guest OSes to trap into
> the hypervisor as necessary.

Yup. Xen is mapped into the top 64MB of virtual address space of every
process (just above the Linux kernel). Segmentation is indeed used to ensure
that:
* user processes can only see their own memory
* guest kernels can see their own memory and user process memory
* Xen can see guest kernel memory and user process memory within the guest

This is a straightforward extension of the way segments are used in vanilla
Linux on x86.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Tobias Hunger wrote:
>
>>Right now Xen is mapped somewhere in top of memory, I am not sure how
>>domains are kept out of there, but I suppose it has to do with segments.
>
>
> As I understand this xen runs in ring0 and pushes the guests kernels one ring
> up into ring1 and then uses traps to allow the guest OSes to trap into the
> hypervisor as necessary.
>
>
>>The good thing about that is that hypercalls are cheap, and in Xen1.x
>>I/O was cheap as well.
>
>
> Cheap where? In dom0 or the VMs?

Both. Assuming that context switching is an expensive operation, and
that most of the cost comes from flushing the TLBs, the current model
has more overhead than a model where the drivers are mapped into the
same address space (though still protected from domU access other that
through hypercalls) as the currently running domain. Xen already
contains magic that allows it to be permanently mapped into running
domains, while still being protected from accidential/malicious access.

I am calling that monolithic, because that is what standard linux does,
and what the old Xen 1.x did. The current model is more like a
microkernel, such as L4 or Mach (though of course a lot more clever, no
offense people ;-))

Apart from perhaps increasing performance, I was saying it would be cool
if the Xen-interface could be provided on platforms such as Windows or
native Linux. People have done similar stuff before (e.g. VMWare vmmon,
the Adeos nanokernel, coLinux), so it should be possible. It seems Xen
would have better chances of world domination that way, if that is a
goal. It would also save some duplication of effort in porting guestOSes
to the different models out there.

Anyway, it was just an idea, and it is likely that with future hardware
the Xen NGIO/microkernel model will finally be overhead-free, vendors
will start writing Xen-VM drivers, and we will all live happily ever
after :-).

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Ronald G. Minnich wrote:
>
> On Thu, 27 Jan 2005, Mark Williamson wrote:
>
>
>>Won't many people running this sort of workload be using specialised
>>network hardware anyhow? A modern cluster interconnect (Infiniband, for
>>instance) will likely support direct IO to user level applications -
>>this could still be done under Xen.
>
>
> that's been our assumption.

I guess so, though the clusters I have seen have all been Ethernet-only,
due to the lower cost. Has anyone had any experience using Xen with this
kind of hardware?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 28 Jan 2005, Jacob Gorm Hansen wrote:

> I guess so, though the clusters I have seen have all been Ethernet-only, due
> to the lower cost. Has anyone had any experience using Xen with this kind of
> hardware?

Adam sulmicki is doing testing of that type of thing now, and we should
have some results in a few weeks on a dual-xeon cluster of about 128
nodes.Part of that will involve a virtual Plan 9 cluster as well on the
same hardware. You just gotta love Xen.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Ronald G. Minnich wrote:
>
> Adam sulmicki is doing testing of that type of thing now, and we should
> have some results in a few weeks on a dual-xeon cluster of about 128
> nodes.Part of that will involve a virtual Plan 9 cluster as well on the
> same hardware. You just gotta love Xen.

I do love Xen. Not to worry ;-)

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

>> I guess so, though the clusters I have seen have all been Ethernet-only, due
>> to the lower cost. Has anyone had any experience using Xen with this kind of
>> hardware?
>
> Adam sulmicki is doing testing of that type of thing now, and we should
> have some results in a few weeks on a dual-xeon cluster of about 128
> nodes.


> Part of that will involve a virtual Plan 9 cluster as well

ah, learn new stuff from the list :-)


> on the same hardware. You just gotta love Xen.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel