Mailing List Archive

> I'm wondering how guest domains are provided access to disks. I suppose
> Domain 0 has direct access to the actual hardware devices, and other domains
> simply see a virtualized block device. However, when a guest domain wants to
> make a request to the disk, how does it get that request to Domain 0 and how
> does Domain 0 actually receive those requests? There appears to be a virtual
> block device driver in drivers/xen/blkfront & blkback. Is this the driver
> used by the guest Domains to access the virtualized devices?

Yes. The blkback driver goes in dom0 (or any other suitably
privileged domain) and is able to export any block device Linux
knows about (e.g. physical partition, LVM volume, loopback file
etc) to it's peer blkfront driver in the guest domain.

> My other question actually pertains to CoW support for disks.
> I noticed that there was some work done on making a CoW driver
> that lived in the XenoLinux kernels. Has this been made
> public? Have there been any attempts to make one that provides
> that functionality in Xen itself?

There are a bunch of CoW options:

There's Bin Ren's CoW driver for Linux 2.4, or you can just use
the standard LVM2 dm-snap stuff in Linux 2.6. The latter
currently doesn't deal well with having many CoW devices, but it
shouldn't be too hard to fix up. (Michael Vrable started looking
at this.)

Since disk space is cheap, it might be better to write a CoW
driver that just uses CoW while it clones the actual disk
content in the background.

If you want CoW at the file system layer, there are a bunch of
different union/stackable/overlay file system kernel drivers for
Linux, but I'm not sure I could actually recommend any of them.

One approach that we've used is a user-space NFS server that
implements CoW semantics. It works OK, but performance isn't as
good as we'd like.

One advantage of CoW schemes is that it should make it easy to
implement a shared buffer cache (as its easy to know when blocks
are identical). This is something we're actively looking in to.

Ian




-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

For this e-mail, I'm assuming we're talking about Xen 2.0...

> I'm wondering how guest domains are provided access to disks. I suppose
> Domain 0 has direct access to the actual hardware devices, and other
> domains simply see a virtualized block device.

All correct.

> However, when a guest
> domain wants to make a request to the disk, how does it get that request to
> Domain 0 and how does Domain 0 actually receive those requests?

When it boots, the unpriv domain sets up sharing of a page of memory between
its frontend driver and the backend driver running in domain 0. The front
and back end drivers do this by sending control messages via Xend. Xend also
binds an interdomain "event channel" between the two domains. This allows
the back and front end drivers to send each other "events" (virtual
interrupts).

Once the shared memory page is set up and the event channel bound, the two are
used for direct communication between the back and front end drivers. Xend
does not have to get involved anymore.

When the frontend wants to request data, it places a descriptor containng
details of the request into the shared memory region. It then sends an
"event" to the backend driver (remember, this is running in dom0), which
checks the shared memory region in response.

Assuming it decides the requests are valid, it will issue requests within the
domain 0 kernel to perform the IO directly into the memory of the unpriv
guest. When the IO has finished, the backend puts a response into the shared
memory page and sends an event to the frontend. The frontend responds to the
virtual interrupt by checking which IO completed and calling the appropriate
completion handlers.

All this happens without Xen knowing about the virtual devices. Xen operates
at the level of shared memory and event channels - it doesn't know or care
that domains are using these as virtual devices.

> There
> appears to be a virtual block device driver in drivers/xen/blkfront &
> blkback. Is this the driver used by the guest Domains to access the
> virtualized devices?

Correct again.

blkfront is the unpriv domain's part of the driver, blkback is the portion
that runs in domain 0 and sends requests to the disk where appropriate.

Note that these are not pretending to be any specific model of real-world
device - they're purely designed for high-performance virtual machine IO.

> My other question actually pertains to CoW support for disks. I noticed
> that there was some work done on making a CoW driver that lived in the
> XenoLinux kernels. Has this been made public?

Since device virtualisation moved into the dom0 kernel and out of Xen, it's
possible to use your favourite CoW driver for vanilla Linux to achieve this.
There's the LVM snapshot facility and, I think, the csnap driver and various
other patches that are out there...

> Have there been any
> attempts to make one that provides that functionality in Xen itself?

The functionality wouldn't be provided in Xen, since Xen itself doesn't manage
devices (that's done by dom0 in Xen 2.0). It wouldn't be a good idea to
bloat the XenLinux backend driver with it anyway - the best idea is for a
generic solution (like LVM, csnap, etc.). Unfortunately LVM and csnap both
have their drawbacks, which have previously been discussed on this list...

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > I'm wondering how guest domains are provided access to disks. I suppose
> > Domain 0 has direct access to the actual hardware devices, and other
> domains
> > simply see a virtualized block device. However, when a guest domain
> wants to
> > make a request to the disk, how does it get that request to Domain 0 and
> how
> > does Domain 0 actually receive those requests? There appears to be a
> virtual
> > block device driver in drivers/xen/blkfront & blkback. Is this the
> driver
> > used by the guest Domains to access the virtualized devices?
>
> Yes. The blkback driver goes in dom0 (or any other suitably
> privileged domain) and is able to export any block device Linux
> knows about (e.g. physical partition, LVM volume, loopback file
> etc) to it's peer blkfront driver in the guest domain.
>

Thanks for the prompt reply.

I'm a bit confused on this point then since the blkfront driver doesn't
appear to make any hypervisor calls to Xen. If it doesn't do this, how does
it communicate requests to the blkback driver that's actually managing the
block device?

Thanks,

David



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> I'm a bit confused on this point then since the blkfront driver doesn't
> appear to make any hypervisor calls to Xen. If it doesn't do this, how
> does it communicate requests to the blkback driver that's actually managing
> the block device?

The block frontend only makes hypercalls in order to send "events" to the
backend driver when new requests are waiting in the shared memory region (the
shared memory region is updated by just assigning to structs in a ring buffer
and incrementing an integer index).

In the network frontend, hypercalls are also used to relinquish buffer pages
to the backend driver but this is not necessary for block devices.

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Thu, 4 Nov 2004, Mark A. Williamson wrote:

> The block frontend only makes hypercalls in order to send "events" to the
> backend driver when new requests are waiting in the shared memory region (the
> shared memory region is updated by just assigning to structs in a ring buffer
> and incrementing an integer index).

I have to say, now that my block frontend is working, that it is a very
easy thing to deal with. Nice design!

And, yes, I'm now booting CPU servers in Plan 9 with network and disk.

Still having some weird Plan 9 issues, but getting there.

ron


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel