Mailing List Archive

Gino LV. Ledesma wrote:

>Hi, list
>
>Has anyone gotten something similar to work? Any ideas / tips /
>comments / suggestions in doing so? I'm tweaking my xen RPM spec file
>to support unionfs and right now just looking at the boot up process
>of getting the domain to mount something else on top of the exported
>root file system (either a file-backed VBD or yet another NFS export).
>
>
This is actually a large part of the paper for this presentation at OLS:

http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117

There's a number of approaches to solving this problem. unionfs would
be ideal but it's a bit unstable. Another approach is to keep certain
directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
read write (/etc, /var/, etc.). This will get you pretty far.

Regards,

Anthony Liguori

>gino ledesma
>
>P.S. If there's anything waaaay simpler than this, I'd really like to
>know. :-) I'm having fun playing with xen but will need to see how
>this setup scales.
>
>_______________________________________________
>Xen-users mailing list
>Xen-users@lists.xensource.com
>http://lists.xensource.com/xen-users
>
>
>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Gino LV. Ledesma wrote:
>
> >Hi, list
> >
> >Has anyone gotten something similar to work? Any ideas / tips /
> >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> >to support unionfs and right now just looking at the boot up process
> >of getting the domain to mount something else on top of the exported
> >root file system (either a file-backed VBD or yet another NFS export).
> >
> >
> This is actually a large part of the paper for this presentation at OLS:
>
> http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
>
> There's a number of approaches to solving this problem. unionfs would
> be ideal but it's a bit unstable. Another approach is to keep certain
> directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> read write (/etc, /var/, etc.). This will get you pretty far.

Thanks for the reply. I guess this is one of the most oftens suggested
approaches. The two goals that I'd like to meet are:
1. Storage flexibility (resize as necessary) -- LVM looks good to go,
though in our setup where we can use NetApps, NFS would be another
approach.
2. Ease of administration / maintenance -- Some of the issues I'm wary
about when going unionfs is that if we make changes to the underlying
filesystem (e.g. OS upgrade) and there are changes on the overlay,
conflicts might occur.

I'll look to doing both -- I'd like to keep things simpler, as there'd
be less chances of breaking things.

Thanks again.

gino ledesma

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I've managed to come up with something pretty neat -- a shared base
image with unionfs providing an overlay for a "delta" filesystem that
allows each host to customize the image to their heart's content while
storing only the changes (deltas). I *love* how xen and open source
work. :-)

xen config exports to the domUs a file-backed master.img which
contains a base root filesystem as /dev/hda1. Then each domU gets
their own LVM-backed logical volumes for root (/dev/hdb1) and swap
(/dev/hdc1). This allows me to grow/shrink partitions as necessary.
And because these partitions only contain changes, they don't use as
much data. I customized the initrd process to union mount the base
root filesystem with the "delta" filesystem and overlay that as root.

This works pretty well, so far -- I'm just trying to make this work
with NFS-exported file systems for maximum flexibility.

Anyway, this may not really be a xen issue, but I thought I'd ask. I
was trying to get several domU's to share a base LVM-backed
filesystem, but when I fire up a second domain using the same fs, it
says that the vbd is already in use. I take it isn't possible to have
multiple mounts of an LVM-backed volume? That's one reason why I had
to use a file-backed root fs above.

Also... As I'm not too sure about this, is it *safe* for a file-backed
root fs to be shared across multiple domUs? I have the initrd mount it
as read-only BUT the xen config has to make it writable. Making it
read only in the config causes the mount process to spit an error
(EXT3 INFO requires write access ...).

-gino

On 5/25/05, Gino LV. Ledesma <gledesma@gmail.com> wrote:
> On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> > Gino LV. Ledesma wrote:
> >
> > >Hi, list
> > >
> > >Has anyone gotten something similar to work? Any ideas / tips /
> > >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> > >to support unionfs and right now just looking at the boot up process
> > >of getting the domain to mount something else on top of the exported
> > >root file system (either a file-backed VBD or yet another NFS export).
> > >
> > >
> > This is actually a large part of the paper for this presentation at OLS:
> >
> > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
> >
> > There's a number of approaches to solving this problem. unionfs would
> > be ideal but it's a bit unstable. Another approach is to keep certain
> > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> > read write (/etc, /var/, etc.). This will get you pretty far.
>
> Thanks for the reply. I guess this is one of the most oftens suggested
> approaches. The two goals that I'd like to meet are:
> 1. Storage flexibility (resize as necessary) -- LVM looks good to go,
> though in our setup where we can use NetApps, NFS would be another
> approach.
> 2. Ease of administration / maintenance -- Some of the issues I'm wary
> about when going unionfs is that if we make changes to the underlying
> filesystem (e.g. OS upgrade) and there are changes on the overlay,
> conflicts might occur.
>
> I'll look to doing both -- I'd like to keep things simpler, as there'd
> be less chances of breaking things.
>
> Thanks again.
>
> gino ledesma
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Anyway, this may not really be a xen issue, but I thought I'd ask. I
> was trying to get several domU's to share a base LVM-backed
> filesystem, but when I fire up a second domain using the same fs, it
> says that the vbd is already in use. I take it isn't possible to have
> multiple mounts of an LVM-backed volume? That's one reason why I had
> to use a file-backed root fs above.

The Xen tools won't let you have multiple mounts to a filesystem unless
they're all read only. This is just a "safety catch" to avoid people
shooting themselves in the foot. If you're Really Sure you want writeable
sharing, stick "w!" as the permission in the config file.

> Also... As I'm not too sure about this, is it *safe* for a file-backed
> root fs to be shared across multiple domUs?

Not if they're mounting it writeable...

> I have the initrd mount it
> as read-only BUT the xen config has to make it writable. Making it
> read only in the config causes the mount process to spit an error
> (EXT3 INFO requires write access ...).

If you mount it read only in the guest, it's probably OK. It's a bit odd that
exporting it read only doesn't work though - we export /usr read only and
then mount it read only in the guest, which seems to work fine.

HTH,
Mark

> -gino
>
> On 5/25/05, Gino LV. Ledesma <gledesma@gmail.com> wrote:
> > On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> > > Gino LV. Ledesma wrote:
> > > >Hi, list
> > > >
> > > >Has anyone gotten something similar to work? Any ideas / tips /
> > > >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> > > >to support unionfs and right now just looking at the boot up process
> > > >of getting the domain to mount something else on top of the exported
> > > >root file system (either a file-backed VBD or yet another NFS export).
> > >
> > > This is actually a large part of the paper for this presentation at
> > > OLS:
> > >
> > > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
> > >
> > > There's a number of approaches to solving this problem. unionfs would
> > > be ideal but it's a bit unstable. Another approach is to keep certain
> > > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> > > read write (/etc, /var/, etc.). This will get you pretty far.
> >
> > Thanks for the reply. I guess this is one of the most oftens suggested
> > approaches. The two goals that I'd like to meet are:
> > 1. Storage flexibility (resize as necessary) -- LVM looks good to go,
> > though in our setup where we can use NetApps, NFS would be another
> > approach.
> > 2. Ease of administration / maintenance -- Some of the issues I'm wary
> > about when going unionfs is that if we make changes to the underlying
> > filesystem (e.g. OS upgrade) and there are changes on the overlay,
> > conflicts might occur.
> >
> > I'll look to doing both -- I'd like to keep things simpler, as there'd
> > be less chances of breaking things.
> >
> > Thanks again.
> >
> > gino ledesma
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Gino,

Mind if I ask a couple questions about your UnionFS setup?

> I've managed to come up with something pretty neat -- a shared base
> image with unionfs providing an overlay for a "delta" filesystem that
> allows each host to customize the image to their heart's content while
> storing only the changes (deltas).

Google turns up a couple projects using that name. Are you using the one
from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a
different one?

> I customized the initrd process to union mount the baseroot filesystem
> with the "delta" filesystem and overlay that as root.

Are there any pitfalls to watch out for in creating this initrd, or is it
pretty straightforward?

Thanks,
mukesh


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/9/05, mukesh agrawal <lists.xensource.com@mukesh.agrawals.org> wrote:
> Google turns up a couple projects using that name. Are you using the one
> from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a
> different one?

Yes, that's the one.

> > I customized the initrd process to union mount the baseroot filesystem
> > with the "delta" filesystem and overlay that as root.
>
> Are there any pitfalls to watch out for in creating this initrd, or is it
> pretty straightforward?

The initrd itself is pretty straightforward, especially if you're
using a file-/physical-/LVM-backed block device. Using NFS gets a
little bit tricky because of the mount procedure, but nothing too
difficult to overcome.

The only issues I'm running into are kernel stack traces during
certain I/O-heavy operations. This has more to do with unionfs I
suppose, rather than the xen kernel itself.

Another one I've looked at recently is mini_fo
<http://www.denx.de/twiki/bin/view/Know/MiniFOHome>, though this
doesn't support overlaying an NFS exported volume yet, and it has some
caching issues I'm looking into.

- gino

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks for your replies. I've used it to start a Wiki page with
information on the various possibilities for copy-on-write functionality.

See http://wiki.xensource.com/xenwiki/COWHowTo.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users