Mailing List Archive

Hi Mike,

i'd like to see your code. :)

Regards
Fabian

2008/11/14 Mike Sun <msun@gatech.edu>

> Hi Nick,
>
> > I was wondering if anyone knows of a way to "checkpoint" without pausing
> the
> > domain? I can't do something like an hourly checkpoint if the domain
> pauses
> > every hour - the users on the domUs would kill me. Is "seamless
> > checkpointing" something that's being developed, something that can be
> done
> > another way, or something I should just give up on now?
>
> I've been working on something for my research that probably addresses
> what you're looking for: low latency/downtime checkpointing. There
> are two ways I know this can be done.
>
> You can modify the live-migration mechanism in Xen to save a
> checkpoint instead of migrating to another host. I don't know why
> this hasn't been provided in the xm save tool, but it's a pretty
> simple hack to get working since the xm migrate and xm save codebase
> is basically the same. If you're interested, I can send patches to
> allow you to do that. Then basically on a checkpoint, pre-copying
> would be used to cut the actual time the domU has to be paused.
>
> The other way that I've implemented for my research project is to use
> copy-on-write. I've implemented a prototype and in the middle of
> writing paper for publication. It's not suited for production use,
> but I'd be glad to contribute the code if there's interest.
>
> Mike
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

Hi Fabian,

We're still working to finish the paper first, but I'll certainly
cleanup and provide the prototype code after we're done.

Thanks,
Mike

On Fri, Nov 14, 2008 at 4:42 AM, Fabian Flägel <fabigant@googlemail.com> wrote:
> Hi Mike,
>
> i'd like to see your code. :)
>
> Regards
> Fabian
>
> 2008/11/14 Mike Sun <msun@gatech.edu>
>>
>> Hi Nick,
>>
>> > I was wondering if anyone knows of a way to "checkpoint" without pausing
>> > the
>> > domain? I can't do something like an hourly checkpoint if the domain
>> > pauses
>> > every hour - the users on the domUs would kill me. Is "seamless
>> > checkpointing" something that's being developed, something that can be
>> > done
>> > another way, or something I should just give up on now?
>>
>> I've been working on something for my research that probably addresses
>> what you're looking for: low latency/downtime checkpointing. There
>> are two ways I know this can be done.
>>
>> You can modify the live-migration mechanism in Xen to save a
>> checkpoint instead of migrating to another host. I don't know why
>> this hasn't been provided in the xm save tool, but it's a pretty
>> simple hack to get working since the xm migrate and xm save codebase
>> is basically the same. If you're interested, I can send patches to
>> allow you to do that. Then basically on a checkpoint, pre-copying
>> would be used to cut the actual time the domU has to be paused.
>>
>> The other way that I've implemented for my research project is to use
>> copy-on-write. I've implemented a prototype and in the middle of
>> writing paper for publication. It's not suited for production use,
>> but I'd be glad to contribute the code if there's interest.
>>
>> Mike
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks for the kind offer, Mike. I too would be interested in seeing
both sets of modifications when they are ready. As well as the paper
when it's finished.

-dwight-

On Thursday 13 November 2008 05:45:56 pm Mike Sun wrote:
> Hi Nick,
>
> > I was wondering if anyone knows of a way to "checkpoint" without
> > pausing the domain? I can't do something like an hourly
> > checkpoint if the domain pauses every hour - the users on the
> > domUs would kill me. Is "seamless checkpointing" something
> > that's being developed, something that can be done another way,
> > or something I should just give up on now?
>
> I've been working on something for my research that probably
> addresses what you're looking for: low latency/downtime
> checkpointing. There are two ways I know this can be done.
>
> You can modify the live-migration mechanism in Xen to save a
> checkpoint instead of migrating to another host. I don't know why
> this hasn't been provided in the xm save tool, but it's a pretty
> simple hack to get working since the xm migrate and xm save
> codebase is basically the same. If you're interested, I can send
> patches to allow you to do that. Then basically on a checkpoint,
> pre-copying would be used to cut the actual time the domU has to
> be paused.
>
> The other way that I've implemented for my research project is to
> use copy-on-write. I've implemented a prototype and in the middle
> of writing paper for publication. It's not suited for production
> use, but I'd be glad to contribute the code if there's interest.
>
> Mike
>
> _______________________________________________
> Xen-research mailing list
> Xen-research@lists.xensource.com
> http://lists.xensource.com/mailman/listinfo/xen-research



_______________________________________________
Xen-research mailing list
Xen-research@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-research

Mike,
I'd also be interested in those patches and that code - something like
an "xm checkpoint" would be great, which sounds like what you're
suggesting or going for in your code there.

Thanks,
Nick


-----Original Message-----
From: Mike Sun <msun@gatech.edu>
To: Fabian Flägel <fabigant@googlemail.com>
Cc: Nick Couchman <Nick.Couchman@seakr.com>, XEN Mailing List
<xen-users@lists.xensource.com>, xen-research@lists.xensource.com
Subject: Re: [Xen-users] xm save -c
Date: Fri, 14 Nov 2008 12:22:38 -0500


Hi Fabian,

We're still working to finish the paper first, but I'll certainly
cleanup and provide the prototype code after we're done.

Thanks,
Mike

On Fri, Nov 14, 2008 at 4:42 AM, Fabian Flägel <fabigant@googlemail.com> wrote:
> Hi Mike,
>
> i'd like to see your code. :)
>
> Regards
> Fabian
>
> 2008/11/14 Mike Sun <msun@gatech.edu>
>>
>> Hi Nick,
>>
>> > I was wondering if anyone knows of a way to "checkpoint" without pausing
>> > the
>> > domain? I can't do something like an hourly checkpoint if the domain
>> > pauses
>> > every hour - the users on the domUs would kill me. Is "seamless
>> > checkpointing" something that's being developed, something that can be
>> > done
>> > another way, or something I should just give up on now?
>>
>> I've been working on something for my research that probably addresses
>> what you're looking for: low latency/downtime checkpointing. There
>> are two ways I know this can be done.
>>
>> You can modify the live-migration mechanism in Xen to save a
>> checkpoint instead of migrating to another host. I don't know why
>> this hasn't been provided in the xm save tool, but it's a pretty
>> simple hack to get working since the xm migrate and xm save codebase
>> is basically the same. If you're interested, I can send patches to
>> allow you to do that. Then basically on a checkpoint, pre-copying
>> would be used to cut the actual time the domU has to be paused.
>>
>> The other way that I've implemented for my research project is to use
>> copy-on-write. I've implemented a prototype and in the middle of
>> writing paper for publication. It's not suited for production use,
>> but I'd be glad to contribute the code if there's interest.
>>
>> Mike
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>



This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.

Hi Mike,

ok, i'll have to wait... :)

Even if your paper is not ready yet, may i ask which COW do you use? I tried
qcow but this doesn't seem to be supported under Ubuntu 8.04 with Xen 3.2.1.

Regards,
Fabian

2008/11/14 Mike Sun <msun@gatech.edu>

> Hi Fabian,
>
> We're still working to finish the paper first, but I'll certainly
> cleanup and provide the prototype code after we're done.
>
> Thanks,
> Mike
>
> On Fri, Nov 14, 2008 at 4:42 AM, Fabian Flägel <fabigant@googlemail.com>
> wrote:
> > Hi Mike,
> >
> > i'd like to see your code. :)
> >
> > Regards
> > Fabian
> >
> > 2008/11/14 Mike Sun <msun@gatech.edu>
> >>
> >> Hi Nick,
> >>
> >> > I was wondering if anyone knows of a way to "checkpoint" without
> pausing
> >> > the
> >> > domain? I can't do something like an hourly checkpoint if the domain
> >> > pauses
> >> > every hour - the users on the domUs would kill me. Is "seamless
> >> > checkpointing" something that's being developed, something that can be
> >> > done
> >> > another way, or something I should just give up on now?
> >>
> >> I've been working on something for my research that probably addresses
> >> what you're looking for: low latency/downtime checkpointing. There
> >> are two ways I know this can be done.
> >>
> >> You can modify the live-migration mechanism in Xen to save a
> >> checkpoint instead of migrating to another host. I don't know why
> >> this hasn't been provided in the xm save tool, but it's a pretty
> >> simple hack to get working since the xm migrate and xm save codebase
> >> is basically the same. If you're interested, I can send patches to
> >> allow you to do that. Then basically on a checkpoint, pre-copying
> >> would be used to cut the actual time the domU has to be paused.
> >>
> >> The other way that I've implemented for my research project is to use
> >> copy-on-write. I've implemented a prototype and in the middle of
> >> writing paper for publication. It's not suited for production use,
> >> but I'd be glad to contribute the code if there's interest.
> >>
> >> Mike
> >>
> >> _______________________________________________
> >> Xen-users mailing list
> >> Xen-users@lists.xensource.com
> >> http://lists.xensource.com/xen-users
> >
> >
>

Qcow also does not currently work with the GPLPV drivers that James is
working on - seems the HVM domUs (in Xen 3.2) don't pass on drive
geometry figures for any tap-based drives.

-Nick


-----Original Message-----
From: Fabian Flägel <fabigant@googlemail.com>
To: Mike Sun <msun@gatech.edu>
Cc: Nick Couchman <Nick.Couchman@seakr.com>, XEN Mailing List
<xen-users@lists.xensource.com>, xen-research@lists.xensource.com
Subject: Re: [Xen-users] xm save -c
Date: Fri, 14 Nov 2008 18:51:08 +0100

Hi Mike,

ok, i'll have to wait... :)

Even if your paper is not ready yet, may i ask which COW do you use? I
tried qcow but this doesn't seem to be supported under Ubuntu 8.04 with
Xen 3.2.1.

Regards,
Fabian

2008/11/14 Mike Sun <msun@gatech.edu>

Hi Fabian,

We're still working to finish the paper first, but I'll
certainly
cleanup and provide the prototype code after we're done.

Thanks,
Mike



On Fri, Nov 14, 2008 at 4:42 AM, Fabian Flägel
<fabigant@googlemail.com> wrote:
> Hi Mike,
>
> i'd like to see your code. :)
>
> Regards
> Fabian
>
> 2008/11/14 Mike Sun <msun@gatech.edu>
>>
>> Hi Nick,
>>
>> > I was wondering if anyone knows of a way to "checkpoint"
without pausing
>> > the
>> > domain? I can't do something like an hourly checkpoint if
the domain
>> > pauses
>> > every hour - the users on the domUs would kill me. Is
"seamless
>> > checkpointing" something that's being developed, something
that can be
>> > done
>> > another way, or something I should just give up on now?
>>
>> I've been working on something for my research that probably
addresses
>> what you're looking for: low latency/downtime checkpointing.
There
>> are two ways I know this can be done.
>>
>> You can modify the live-migration mechanism in Xen to save a
>> checkpoint instead of migrating to another host. I don't
know why
>> this hasn't been provided in the xm save tool, but it's a
pretty
>> simple hack to get working since the xm migrate and xm save
codebase
>> is basically the same. If you're interested, I can send
patches to
>> allow you to do that. Then basically on a checkpoint,
pre-copying
>> would be used to cut the actual time the domU has to be
paused.
>>
>> The other way that I've implemented for my research project
is to use
>> copy-on-write. I've implemented a prototype and in the
middle of
>> writing paper for publication. It's not suited for
production use,
>> but I'd be glad to contribute the code if there's interest.
>>
>> Mike
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>





This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.

The modification to do a checkpoint using the live migration mechanism
of pre-copying is a trivial one and just something we're benchmarking
against with our CoW solution. Here's a patch that changes 'xm save
-c' to do the checkpoint using the live migration mechanism instead of
the default way. Word of warning: it's a hack and I haven't made the
changes necessary so that the checkpoint file won't contain the pages
copied during the copy rounds so the checkpoint file will be much
larger than necessary.

I'm not sure why a checkpoint with pre-copying mechanism hasn't been
enabled in Xen yet as the necessary code is all there.

Cheers,
Mike

diff -r dca3b5e459e3 tools/python/xen/xend/XendDomain.py
--- a/tools/python/xen/xend/XendDomain.py Wed Sep 17 15:46:05 2008 +0100
+++ b/tools/python/xen/xend/XendDomain.py Fri Nov 14 13:28:39 2008 -0500
@@ -1333,7 +1333,8 @@
oflags |= os.O_LARGEFILE
fd = os.open(dst, oflags)
try:
- XendCheckpoint.save(fd, dominfo, False, False, dst,
+ live = True
+ XendCheckpoint.save(fd, dominfo, False, live, dst,
checkpoint=checkpoint)
except Exception, e:
os.close(fd)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Even if your paper is not ready yet, may i ask which COW do you use? I tried
> qcow but this doesn't seem to be supported under Ubuntu 8.04 with Xen 3.2.1.

I think there's some confusion. Are you interested in CoW for the
disk or for the memory checkpoint? QCOW is a disk format for disk
checkpointing. The earlier question by Nick seemed more interested in
doing a seamless memory checkpoint of a running VM.

_______________________________________________
Xen-research mailing list
Xen-research@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-research

Whoops, forgot to mention this is a patch for Xen 3.2.2, though I'm
sure the changes can be applied to unstable.

On Fri, Nov 14, 2008 at 1:35 PM, Mike Sun <msun@gatech.edu> wrote:
> The modification to do a checkpoint using the live migration mechanism
> of pre-copying is a trivial one and just something we're benchmarking
> against with our CoW solution. Here's a patch that changes 'xm save
> -c' to do the checkpoint using the live migration mechanism instead of
> the default way. Word of warning: it's a hack and I haven't made the
> changes necessary so that the checkpoint file won't contain the pages
> copied during the copy rounds so the checkpoint file will be much
> larger than necessary.
>
> I'm not sure why a checkpoint with pre-copying mechanism hasn't been
> enabled in Xen yet as the necessary code is all there.
>
> Cheers,
> Mike
>
> diff -r dca3b5e459e3 tools/python/xen/xend/XendDomain.py
> --- a/tools/python/xen/xend/XendDomain.py Wed Sep 17 15:46:05 2008 +0100
> +++ b/tools/python/xen/xend/XendDomain.py Fri Nov 14 13:28:39 2008 -0500
> @@ -1333,7 +1333,8 @@
> oflags |= os.O_LARGEFILE
> fd = os.open(dst, oflags)
> try:
> - XendCheckpoint.save(fd, dominfo, False, False, dst,
> + live = True
> + XendCheckpoint.save(fd, dominfo, False, live, dst,
> checkpoint=checkpoint)
> except Exception, e:
> os.close(fd)

_______________________________________________
Xen-research mailing list
Xen-research@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-research

On Friday, 14 November 2008 at 13:35, Mike Sun wrote:
> The modification to do a checkpoint using the live migration mechanism
> of pre-copying is a trivial one and just something we're benchmarking
> against with our CoW solution. Here's a patch that changes 'xm save
> -c' to do the checkpoint using the live migration mechanism instead of
> the default way. Word of warning: it's a hack and I haven't made the
> changes necessary so that the checkpoint file won't contain the pages
> copied during the copy rounds so the checkpoint file will be much
> larger than necessary.
>
> I'm not sure why a checkpoint with pre-copying mechanism hasn't been
> enabled in Xen yet as the necessary code is all there.

I think it's mostly just that the semantics are a bit weird -- you
don't have a great idea what point in guest time corresponds to the
checkpoint you'll end up with. But we've happily that mode here for
things like coordinated checkpoints.


_______________________________________________
Xen-research mailing list
Xen-research@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-research