Mailing List Archive

> Hello
>
> Some time ago Michael Vrable talked about copy-on-write memory to enable
> large numbers of nearly identical machines to run on the same physical
> hardware. Is this a feasible proposition? I asked a few days ago in the
> original thread, but no-one seems to have noticed. It seems to me that
> it would be a very significant feature to offer.

It's useful for honeypots and other situations where you want
very large numbers of VMs, but isn't generally a huge win.

When I was doing the live migration work I recorded fingerprints
of all VM pages that were on several systems, and didn't find a
whole lot of commonality between VMs. CoW sharing of memory is on
the list of features we'd like to add to Xen, but implementing
the shared buffer cache is rather higher on the todo list.

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Ian Pratt wrote:
>>Hello
>>
>>Some time ago Michael Vrable talked about copy-on-write memory to enable
>>large numbers of nearly identical machines to run on the same physical
>>hardware. Is this a feasible proposition? I asked a few days ago in the
>>original thread, but no-one seems to have noticed. It seems to me that
>>it would be a very significant feature to offer.
>
> It's useful for honeypots and other situations where you want
> very large numbers of VMs, but isn't generally a huge win.
>
> When I was doing the live migration work I recorded fingerprints
> of all VM pages that were on several systems, and didn't find a
> whole lot of commonality between VMs.

I thought the idea would be to have a 'clone' primitive that would work
in a similar way to Unix fork? The clone would start off sharing all
pages, although a lot of them would get copied fairly quickly. Does
anyone have an idea of what proportion or number of pages in a Linux
system stay constant (and are not paged out) after boot?

--
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 25 Oct 2004, David Hopwood wrote:

> I thought the idea would be to have a 'clone' primitive that would work
> in a similar way to Unix fork? The clone would start off sharing all
> pages, although a lot of them would get copied fairly quickly. Does


yes but ... if most or almost most pages get copied, COW is a loss. Try to
find that old paper somebody wrote about "bovaphobic" behaviour --
interesting numbers in there.

COW is not always a good thing.

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> I thought the idea would be to have a 'clone' primitive that would work
> in a similar way to Unix fork? The clone would start off sharing all
> pages, although a lot of them would get copied fairly quickly. Does
> anyone have an idea of what proportion or number of pages in a Linux
> system stay constant (and are not paged out) after boot?

We've already done some work on implementing very lightweight VM
checkpoints, for purposes of debugging and s/w fault
tolerance. 'VM clone' is on the todo list, but I think it'll be a
while before this makes it into a stable release.

Ian



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote:
> Some time ago Michael Vrable talked about copy-on-write memory to enable
> large numbers of nearly identical machines to run on the same physical
> hardware. Is this a feasible proposition? I asked a few days ago in the
> original thread, but no-one seems to have noticed. It seems to me that
> it would be a very significant feature to offer.

I'm still around, though a bit slow to respond.

Ian mentioned that honeypots are one the scenarios where copy-on-write
sharing of memory makes a lot of sense; that's the reason that I'm
trying implement it. I hope, of course, that if I do manage to get
something working it would be useful in other scenarios as well.

My current status: I've been working first on using shadow page tables
to hide the actual machine addresses of pages from domains. I'm only
partway through doing that, and may yet go back and change how I'm
approaching that (I'm still learning bits about how Xen works as I'm
going through to modify the code; I'm also thinking now of changes that
would be good to make copy-on-write easier later). I'm trying to write
up some of my design thoughts; if anyone here is interested or would
like to offer comments, I can post that here.

David mentioned implementing a clone() for VMs; that is something that
I'd also like to try out. (It should be possible to test that after
only the first step of hiding machine addresses, before copy-on-write is
implemented, though copy-on-write will make it efficient.)

If someone else beats me to getting this implemented, I suppose I can't
complain. But if this is an area that isn't getting much current work,
and I'm able to contribute (a hopefully clean implementation that others
wouldn't mind using), that would be great.

--Michael Vrable


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

I hadn't been thinking of the honeypot scenario myself, but of the plain
old 'more virtual computers than you would have thought possible' scene.
From Ian's account of it there's less scope for that than I had imagined.

I can be heavy slow myself ('heavy' is apparently Glaswgian slang for
'very' - think 'heavy light') - especially when immersed in the
technical undergrowth.

Regards
Peri




Michael Vrable wrote:

>On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote:
>
>
>>Some time ago Michael Vrable talked about copy-on-write memory to enable
>>large numbers of nearly identical machines to run on the same physical
>>hardware. Is this a feasible proposition? I asked a few days ago in the
>>original thread, but no-one seems to have noticed. It seems to me that
>>it would be a very significant feature to offer.
>>
>>
>
>I'm still around, though a bit slow to respond.
>
>Ian mentioned that honeypots are one the scenarios where copy-on-write
>sharing of memory makes a lot of sense; that's the reason that I'm
>trying implement it. I hope, of course, that if I do manage to get
>something working it would be useful in other scenarios as well.
>
>My current status: I've been working first on using shadow page tables
>to hide the actual machine addresses of pages from domains. I'm only
>partway through doing that, and may yet go back and change how I'm
>approaching that (I'm still learning bits about how Xen works as I'm
>going through to modify the code; I'm also thinking now of changes that
>would be good to make copy-on-write easier later). I'm trying to write
>up some of my design thoughts; if anyone here is interested or would
>like to offer comments, I can post that here.
>
>David mentioned implementing a clone() for VMs; that is something that
>I'd also like to try out. (It should be possible to test that after
>only the first step of hiding machine addresses, before copy-on-write is
>implemented, though copy-on-write will make it efficient.)
>
>If someone else beats me to getting this implemented, I suppose I can't
>complain. But if this is an area that isn't getting much current work,
>and I'm able to contribute (a hopefully clean implementation that others
>wouldn't mind using), that would be great.
>
>--Michael Vrable
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>Use IT products in your business? Tell us what you think of them. Give us
>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>
>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, Oct 26, 2004 at 09:15:05AM +0100, Peri Hankey wrote:
> I hadn't been thinking of the honeypot scenario myself, but of the plain
> old 'more virtual computers than you would have thought possible' scene.
> From Ian's account of it there's less scope for that than I had imagined.
>
> I can be heavy slow myself ('heavy' is apparently Glaswgian slang for
> 'very' - think 'heavy light') - especially when immersed in the
> technical undergrowth.

"Memory Resource Management in VMware ESX Server" by Carl Waldspurger
has some numbers on memory sharing between VMs:
http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
(This paper is also cited in the Xen SOSP 2003 paper.)

Sharing between identical machines is good, though sharing between
dissimilar machines is not as high as I remembered (copy-on-write
sharing is able to save around 7-30% of memory in real-world testing).

--Michael Vrable


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> "Memory Resource Management in VMware ESX Server" by Carl Waldspurger
> has some numbers on memory sharing between VMs:
> http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
> (This paper is also cited in the Xen SOSP 2003 paper.)
>
> Sharing between identical machines is good, though sharing between
> dissimilar machines is not as high as I remembered (copy-on-write
> sharing is able to save around 7-30% of memory in real-world testing).

As I recall, Carl's results were on Windows, which might explain
why I was seeing rather less. Linux resident set sizes tend to be
smaller, and there's generally rather more diversity in the
install base (at least around here).

Ian


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, Oct 26, 2004 at 08:43:30PM +0100, Ian Pratt wrote:
> > "Memory Resource Management in VMware ESX Server" by Carl Waldspurger
> > has some numbers on memory sharing between VMs:
> > http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
> > (This paper is also cited in the Xen SOSP 2003 paper.)
> >
> > Sharing between identical machines is good, though sharing between
> > dissimilar machines is not as high as I remembered (copy-on-write
> > sharing is able to save around 7-30% of memory in real-world testing).
>
> As I recall, Carl's results were on Windows, which might explain
> why I was seeing rather less. Linux resident set sizes tend to be
> smaller, and there's generally rather more diversity in the
> install base (at least around here).

Both were tested in that paper; Windows was at the high end of the scale
and Linux at the low end:

Reproducing Figure 5 from the paper in slightly abbreviated form:

Guests % Shared % Reclaimed
10 x WinNT 42.9 32.9
9 x Linux 29.2 18.7
5 x Linux 10.0 7.2

"% Shared" is the fraction of pages which are shared; "% Reclaimed"
measures how much memory sharing the pages is able to save.

Three systems is still a limited test set.

--Michael Vrable


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Michael Vrable wrote:

> My current status: I've been working first on using shadow page tables
> to hide the actual machine addresses of pages from domains. I'm only
> partway through doing that, and may yet go back and change how I'm
> approaching that (I'm still learning bits about how Xen works as I'm
> going through to modify the code; I'm also thinking now of changes that
> would be good to make copy-on-write easier later). I'm trying to write
> up some of my design thoughts; if anyone here is interested or would
> like to offer comments, I can post that here.
>
> David mentioned implementing a clone() for VMs; that is something that
> I'd also like to try out. (It should be possible to test that after
> only the first step of hiding machine addresses, before copy-on-write is
> implemented, though copy-on-write will make it efficient.

What about having a special variant of the pte-update operation, which
takes as input a pointer to a pte-entry, and will change that entry into
a read-only mapping of the same page contents, though perhaps with a
different frame number in the pte.

Then Xen can lookup the page contents in a global hashtable or binary
tree (which might be faster, because you don't need to scan the entire
page contents unless you have a match), and see if there is a matching
copy somewhere. If there is, it can decrease a refcount for the page
provided by the user, change the mapping to point to the matching page,
and free the user's page if the refcount has reached zero.

In this way, domains become responsible for sharing memory across the
system, but only pages that are likely candidates for sharing (kernel
text, user process text) are actually scanned, and you don't get the
overhead of shadow page tables.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> What about having a special variant of the pte-update operation, which
> takes as input a pointer to a pte-entry, and will change that entry into
> a read-only mapping of the same page contents, though perhaps with a
> different frame number in the pte.

I'm still not convinced we wouldn't get most of the benefit by
just sharing pages read from the block/file system, and hence
avoiding all the hashing.

Collecting this data is actually very easy -- its just a case of
coming up with a few "realistic usage scenarios" in which to
collect the data. We'd be happy to work with someone to do
this...

Ian


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Check the s/390 port of the kernel for the XIP2 filesystem. It exploits
the z/VM shared memory system to make a read-only ext2 like filesystem
that is shared amonst all the guests - and uses execute-in-place. Any
binaries on the xip2 filesystem aren't cached into local shared memory,
they're just mapped across.

This lets an admin build explicit shared systems, does shared ro-memory
nicely, etc. Might be worth looking into implementing similar in Xen.

Apologies if this was already discussed/noticed,

-m

On Tue, Nov 02, 2004 at 09:33:00PM +0000, Ian Pratt wrote:
>
> > What about having a special variant of the pte-update operation, which
> > takes as input a pointer to a pte-entry, and will change that entry into
> > a read-only mapping of the same page contents, though perhaps with a
> > different frame number in the pte.
>
> I'm still not convinced we wouldn't get most of the benefit by
> just sharing pages read from the block/file system, and hence
> avoiding all the hashing.
>
> Collecting this data is actually very easy -- its just a case of
> coming up with a few "realistic usage scenarios" in which to
> collect the data. We'd be happy to work with someone to do
> this...
>
> Ian
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel