Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Xen>
  3. Devel
PV drivers on HVM using Xen 4.1.1
alex at alex
Oct 27, 2011, 5:56 AM
Post #1 of 22 (1406 views)
Permalink
I think I've found one reason why we can't get PV block drivers on HVM
domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
domU kernel.

We are using a line like:

disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]

We are using effectively a standard 3.0 kernel. Config options including
the word XEN are below.

We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package
does not contain blktapctrl at all (which would explain why it doesn't
start). Do we need this?

It has been suggested that we don't need this, but we do need a kernel
module that provides blktap.

http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html

suggests these might be queued for the non-existent "2.6.40", but
it's suggested these aren't in 3.0.

http://wiki.xensource.com/xenwiki/XAPI_on_debian

suggests there are dmks modules available, but that blktap is currenly
only 32 bit. Can that be correct?

Isn't this what blkback does?

domU config is below. Note the very same VM with the same config file
works just fine on xen 3.3.1. This domU is Centos 2.6.18 with
unmodified_drivers xenlinux type kernel (as supplied by Centos). Every
other kernel we've tried does the same, save that modern ones also
unplug the emulated devices so no disks appear as well.

PV nics work fine.

--
Alex Bligh

root@xen4:/home/ubuntu/kernel# fgrep XEN /boot/config-3.0.0-12-server
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_DEBUG is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=m
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=m
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_XEN_PLATFORM_PCI=m
CONFIG_SWIOTLB_XEN=y

root@xen4:/home/ubuntu/kernel# lsmod | fgrep -i xen
xen_netfront 26671 0
xen_blkfront 26261 0
xen_wdt 13525 0
xen_gntalloc 13321 0
xenbus_probe_frontend 13232 2 xen_netfront,xen_blkfront,[permanent]
xen_gntdev 17676 2
netxen_nic 97283 0
xen_netback 27854 0 [permanent]
xen_blkback 23177 0 [permanent]
xen_evtchn 13172 5
xenfs 18311 1

Guest config looks like this, though we've used xvda and hda

root@xen4:/home/ubuntu/kernel# cat !$
cat /etc/xen/centos5-pvd
name = "centos5-pvd-xen4"
uuid = "6d72aff9-8f53-beb7-c71c-89e72b93193e"
maxmem = 512
memory = 512
vcpus = 2
builder = "hvm"
#kernel = "/usr/lib/xen/boot/hvmloader"
boot = "c"
pae = 1
acpi = 1
apic = 1
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
#device_model = "/usr/lib64/xen/bin/pv-qemu-dm"
#device_model = "/usr/lib/xen/bin/stubdom-dm"
#sdl = 0
#vnc = 1
vnclisten="0.0.0.0"
keymap="en-gb"
#vncunused = 1
#vncdisplay = "3"
vfb = [ "type=vnc,vncunused=1,keymap=en-gb" ]
disk = [ "file:/root/Iain2011/centos-pvd.img,xvda,w" ]
vif = [
"mac=00:16:3e:00:a5:58,bridge=br0,type=foo","mac=00:16:3e:00:a5:58,bridge=br0,type=ioemu"
]
serial = "pty"




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
konrad.wilk at oracle
Oct 27, 2011, 6:35 AM
Post #2 of 22 (1388 views)
Permalink
On Thu, Oct 27, 2011 at 01:56:41PM +0100, Alex Bligh wrote:
> I think I've found one reason why we can't get PV block drivers on HVM
> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
> domU kernel.
>
> We are using a line like:
>
> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
>
> We are using effectively a standard 3.0 kernel. Config options including
> the word XEN are below.
>
> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package
> does not contain blktapctrl at all (which would explain why it doesn't
> start). Do we need this?
>
> It has been suggested that we don't need this, but we do need a kernel
> module that provides blktap.
>
> http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html
>
> suggests these might be queued for the non-existent "2.6.40", but
> it's suggested these aren't in 3.0.

Right, they are a no-go. There is a
>
> http://wiki.xensource.com/xenwiki/XAPI_on_debian
>
> suggests there are dmks modules available, but that blktap is currenly
> only 32 bit. Can that be correct?

There is a 64-bit (and 32-bit) version on Daniel's git tree:

git://xenbits.xensource.com/people/dstodden/linux.git

.. but the deal is that it is unmaintained (Daniel left Citrix).

Thought interestingly .. he made a version of it where all of the fiddling
with the generic code has been removed. Neat.

>
> Isn't this what blkback does?

Blkback can't handle files - it can only handle block devices.
>
> domU config is below. Note the very same VM with the same config file
> works just fine on xen 3.3.1. This domU is Centos 2.6.18 with
> unmodified_drivers xenlinux type kernel (as supplied by Centos). Every
> other kernel we've tried does the same, save that modern ones also
> unplug the emulated devices so no disks appear as well.
>
> PV nics work fine.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 6:44 AM
Post #3 of 22 (1393 views)
Permalink
On Thu, 2011-10-27 at 13:56 +0100, Alex Bligh wrote:
> I think I've found one reason why we can't get PV block drivers on HVM
> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
> domU kernel.
>
> We are using a line like:
>
> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
>
> We are using effectively a standard 3.0 kernel. Config options including
> the word XEN are below.
>
> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> start in Xen3.

Are you changing dom0 kernel as well as hypervisor when you say "3" and
"4.1.1" or is the dom0 kernel constant?

> In fact as far as I can tell the Ubuntu Xen4 package
> does not contain blktapctrl at all (which would explain why it doesn't
> start). Do we need this?

AFAIK tap:aio: usually requires blktap (both kernel side and userspace).
There is also a block backend in qemu which can be by xl used under some
circumstances when blktap is not available but the same is not true of
xend.

> It has been suggested that we don't need this, but we do need a kernel
> module that provides blktap.
>
> http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html
>
> suggests these might be queued for the non-existent "2.6.40", but
> it's suggested these aren't in 3.0.
>
> http://wiki.xensource.com/xenwiki/XAPI_on_debian
>
> suggests there are dmks modules available, but that blktap is currenly
> only 32 bit. Can that be correct?

XCP (from whence that blktap version comes) is only developed/test using
32 bit. Taht's not to say 64 bit doesn't work, but it might not.

> Isn't this what blkback does?

blkback can only export raw block devices to guests.

blktap can be used to export more "structured" image types, such as vhd.

A raw img like you have cannot be directly handled by blkback (since it
is a file and not a block device). You can consider it a dengenerate
case of a structured image so blktap can deal with it. A raw image can
also be mounted with a loop back device which itself can be exported via
blkback. Some toolstacks (xend) can do this automatically (but won't for
a tap:aio: AFAIK) but xl will not.

Unfortunately blktap (at least in its current form) is not suitable to
go upstream. If someone wants to step up and implement the blkring
protocol in userspace (i.e. do away with the kernel component by merging
that bit into the tapdisk process itself) then that would be very much
appreciated.

> domU config is below. Note the very same VM with the same config file
> works just fine on xen 3.3.1. This domU is Centos 2.6.18 with
> unmodified_drivers xenlinux type kernel (as supplied by Centos). Every
> other kernel we've tried does the same, save that modern ones also
> unplug the emulated devices so no disks appear as well.
>
> PV nics work fine.
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
stefano.stabellini at eu
Oct 27, 2011, 6:54 AM
Post #4 of 22 (1383 views)
Permalink
On Thu, 27 Oct 2011, Alex Bligh wrote:
> I think I've found one reason why we can't get PV block drivers on HVM
> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
> domU kernel.
>
> We are using a line like:
>
> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]

If this is an HVM guest, you should use hda here because you want to
make sure that an emulated IDE disk is created for you as well.
Also if /tmp/centos-pvd.img is a raw file, you might as well use file:
rather than tap:aio.


> We are using effectively a standard 3.0 kernel. Config options including
> the word XEN are below.
>
> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package
> does not contain blktapctrl at all (which would explain why it doesn't
> start). Do we need this?
>
> It has been suggested that we don't need this, but we do need a kernel
> module that provides blktap.
>
> http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html
>
> suggests these might be queued for the non-existent "2.6.40", but
> it's suggested these aren't in 3.0.
>
> http://wiki.xensource.com/xenwiki/XAPI_on_debian
>
> suggests there are dmks modules available, but that blktap is currenly
> only 32 bit. Can that be correct?
>
> Isn't this what blkback does?

blkback only works with block devices. You can still use blkback if you
configure a loop device:

losetup /dev/loop0 /tmp/centos-pvd.img

and then:

disk = [ "phy:/dev/loop0,hda,w" ]

However is not going to be very fast unfortunately.

As an alternative you could download and configure upstream qemu with
linux aio and then use it as device model or just block backend but
unfortunately it needs some hacking on 4.1 (on 4.2 is semi automated
now).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 7:04 AM
Post #5 of 22 (1389 views)
Permalink
Ian,

--On 27 October 2011 14:44:49 +0100 Ian Campbell <Ian.Campbell@citrix.com>
wrote:


>> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
>> start in Xen3.
>
> Are you changing dom0 kernel as well as hypervisor when you say "3" and
> "4.1.1" or is the dom0 kernel constant?

No, the Xen3 kernel is 2.6.18 Centos, and the Xen 4 kernel is 3.0
Ubuntu Oneiric.

>> In fact as far as I can tell the Ubuntu Xen4 package
>> does not contain blktapctrl at all (which would explain why it doesn't
>> start). Do we need this?
>
> AFAIK tap:aio: usually requires blktap (both kernel side and userspace).
> There is also a block backend in qemu which can be by xl used under some
> circumstances when blktap is not available but the same is not true of
> xend.

We are using xl (though have tried xend).

> XCP (from whence that blktap version comes) is only developed/test using
> 32 bit. Taht's not to say 64 bit doesn't work, but it might not.

OK. The idea here is a production xen4.1 system, so "might not work"
sounds like a bad idea :-)

>> Isn't this what blkback does?
>
> blkback can only export raw block devices to guests.

We will in production use raw block devices, so that's ok. We were
just using files for testing. Is the same restriction there is
Xen3.3, because it works there (possibly for the reason below).

> A raw image can
> also be mounted with a loop back device which itself can be exported via
> blkback. Some toolstacks (xend) can do this automatically (but won't for
> a tap:aio: AFAIK) but xl will not.

I think xl /must/ be doing this, or it is difficult to explain the
behaviour below.

We discovered Ubuntu's Xen 4.1 hypervisor package does not contain
blktapctrl, so we copied that across from a build from source, plus some
libraries, and did an mknod on /dev/xen/blktap0 (which it appeared to want
to open). pvdrivers then appeared in the guest. But we then reversed
everything we'd done (we think), and pvdrivers continued to appear. We are
reinstalling to find out exactly what it was that changed.

I must admit to remaining almost totally confused as to how this is
meant to work, but thanks for your help thus far!

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 7:10 AM
Post #6 of 22 (1419 views)
Permalink
On Thu, 2011-10-27 at 15:04 +0100, Alex Bligh wrote:
> Ian,
>
> --On 27 October 2011 14:44:49 +0100 Ian Campbell <Ian.Campbell@citrix.com>
> wrote:
>
>
> >> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> >> start in Xen3.
> >
> > Are you changing dom0 kernel as well as hypervisor when you say "3" and
> > "4.1.1" or is the dom0 kernel constant?
>
> No, the Xen3 kernel is 2.6.18 Centos, and the Xen 4 kernel is 3.0
> Ubuntu Oneiric.

OK, that is important information now that Xen is not supplied with a
particular kernel.

> >> In fact as far as I can tell the Ubuntu Xen4 package
> >> does not contain blktapctrl at all (which would explain why it doesn't
> >> start). Do we need this?
> >
> > AFAIK tap:aio: usually requires blktap (both kernel side and userspace).
> > There is also a block backend in qemu which can be by xl used under some
> > circumstances when blktap is not available but the same is not true of
> > xend.
>
> We are using xl (though have tried xend).

Good to know.

> > XCP (from whence that blktap version comes) is only developed/test using
> > 32 bit. Taht's not to say 64 bit doesn't work, but it might not.
>
> OK. The idea here is a production xen4.1 system, so "might not work"
> sounds like a bad idea :-)
>
> >> Isn't this what blkback does?
> >
> > blkback can only export raw block devices to guests.
>
> We will in production use raw block devices, so that's ok. We were
> just using files for testing. Is the same restriction there is
> Xen3.3, because it works there (possibly for the reason below).

The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a
big factor. blkback.ko is in both so you should be ok. For testing you
could try using loopback like Stefano suggests.

> > A raw image can
> > also be mounted with a loop back device which itself can be exported via
> > blkback. Some toolstacks (xend) can do this automatically (but won't for
> > a tap:aio: AFAIK) but xl will not.
>
> I think xl /must/ be doing this, or it is difficult to explain the
> behaviour below.

Indeed, I can't explain it either. I think I would expect xl to fall
back to the qemu based backend if blktap wasn't available. I _think_
that functionality was backported to the qemu-xen in 4.1, but I'm not
sure.

> We discovered Ubuntu's Xen 4.1 hypervisor package does not contain
> blktapctrl, so we copied that across from a build from source, plus some
> libraries, and did an mknod on /dev/xen/blktap0 (which it appeared to want
> to open). pvdrivers then appeared in the guest. But we then reversed
> everything we'd done (we think), and pvdrivers continued to appear. We are
> reinstalling to find out exactly what it was that changed.
>
> I must admit to remaining almost totally confused as to how this is
> meant to work, but thanks for your help thus far!
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 7:11 AM
Post #7 of 22 (1382 views)
Permalink
--On 27 October 2011 14:54:48 +0100 Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:

>> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
>
> If this is an HVM guest,

It is

> you should use hda here because you want to
> make sure that an emulated IDE disk is created for you as well.

On Xen3.3 (and apparently on Xen 4) this happens anyway

> Also if /tmp/centos-pvd.img is a raw file, you might as well use file:
> rather than tap:aio.

There seems to be some doubt (see Ian's message) about whether this
changes the backend driver that is used. The final deployment application
is tap:aio with a block device, so that's why we're doing this.

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 7:16 AM
Post #8 of 22 (1381 views)
Permalink
--On 27 October 2011 15:10:45 +0100 Ian Campbell <Ian.Campbell@citrix.com>
wrote:

>> We will in production use raw block devices, so that's ok. We were
>> just using files for testing. Is the same restriction there is
>> Xen3.3, because it works there (possibly for the reason below).
>
> The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a
> big factor. blkback.ko is in both so you should be ok. For testing you
> could try using loopback like Stefano suggests.

So, with apologies for being dense, what disk line should we be using
to use blkback.ko? I'm assuming this is faster than the qemu based
backend? (i.e. what you think xl may fall back to if blktap is not
available)

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
stefano.stabellini at eu
Oct 27, 2011, 7:19 AM
Post #9 of 22 (1379 views)
Permalink
On Thu, 27 Oct 2011, Alex Bligh wrote:
> > you should use hda here because you want to
> > make sure that an emulated IDE disk is created for you as well.
>
> On Xen3.3 (and apparently on Xen 4) this happens anyway
>

Yes, but I wouldn't rely on that: if you specify xvda it means you want
*only* a pv disk. However if you don't have any IDE disks configured,
qemu realizes that you made a mistake in the config and setup one for
you anyway.


> > Also if /tmp/centos-pvd.img is a raw file, you might as well use file:
> > rather than tap:aio.
>
> There seems to be some doubt (see Ian's message) about whether this
> changes the backend driver that is used. The final deployment application
> is tap:aio with a block device, so that's why we're doing this.

If you are using XL, no matter if you specify tap:aio or file:, you are
going to get qemu as disk backend if you are missing blktap.
There is nothing wrong with that, except that qemu in 4.1 doesn't
support linux aio so the performances are not very good. I am not sure
which one is better: blkback on a loop device or qemu without linux aio,
they are both rather slow.

In order to make it fast you can:

- use a dom0 kernel that provides blktap;

- use LVM with blkback;

- use upstream qemu with linux aio as device model and/or block backend.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 7:19 AM
Post #10 of 22 (1386 views)
Permalink
On Thu, 2011-10-27 at 15:16 +0100, Alex Bligh wrote:
>
> --On 27 October 2011 15:10:45 +0100 Ian Campbell <Ian.Campbell@citrix.com>
> wrote:
>
> >> We will in production use raw block devices, so that's ok. We were
> >> just using files for testing. Is the same restriction there is
> >> Xen3.3, because it works there (possibly for the reason below).
> >
> > The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a
> > big factor. blkback.ko is in both so you should be ok. For testing you
> > could try using loopback like Stefano suggests.
>
> So, with apologies for being dense, what disk line should we be using
> to use blkback.ko? I'm assuming this is faster than the qemu based
> backend? (i.e. what you think xl may fall back to if blktap is not
> available)

I think just "/dev/something,,hda" (or xvda)

The disk syntax is documented in docs/misc/xl-disk-configuration.txt,
all this stuff with phy: and tap:aoi: prefixes is really legacy for
compatibility with xend, with xend you might have used
"phy:/dev/something...." IIRC.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 7:22 AM
Post #11 of 22 (1382 views)
Permalink
On Thu, 2011-10-27 at 15:11 +0100, Alex Bligh wrote:
>
> --On 27 October 2011 14:54:48 +0100 Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>
> >> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
> >
> > If this is an HVM guest,
>
> It is
>
> > you should use hda here because you want to
> > make sure that an emulated IDE disk is created for you as well.
>
> On Xen3.3 (and apparently on Xen 4) this happens anyway
>
> > Also if /tmp/centos-pvd.img is a raw file, you might as well use file:
> > rather than tap:aio.
>
> There seems to be some doubt (see Ian's message) about whether this
> changes the backend driver that is used. The final deployment application
> is tap:aio with a block device, so that's why we're doing this.

tap:aio on a raw block device is not a useful combination.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 7:29 AM
Post #12 of 22 (1377 views)
Permalink
On Thu, 2011-10-27 at 15:19 +0100, Stefano Stabellini wrote:
> On Thu, 27 Oct 2011, Alex Bligh wrote:
> > > you should use hda here because you want to
> > > make sure that an emulated IDE disk is created for you as well.
> >
> > On Xen3.3 (and apparently on Xen 4) this happens anyway
> >
>
> Yes, but I wouldn't rely on that: if you specify xvda it means you want
> *only* a pv disk. However if you don't have any IDE disks configured,
> qemu realizes that you made a mistake in the config and setup one for
> you anyway.

Specifically _some_ qemu's realize...

> > > Also if /tmp/centos-pvd.img is a raw file, you might as well use file:
> > > rather than tap:aio.
> >
> > There seems to be some doubt (see Ian's message) about whether this
> > changes the backend driver that is used. The final deployment application
> > is tap:aio with a block device, so that's why we're doing this.
>
> If you are using XL, no matter if you specify tap:aio or file:, you are
> going to get qemu as disk backend if you are missing blktap.
> There is nothing wrong with that, except that qemu in 4.1 doesn't
> support linux aio so the performances are not very good. I am not sure
> which one is better: blkback on a loop device or qemu without linux aio,
> they are both rather slow.
>
> In order to make it fast you can:
>
> - use a dom0 kernel that provides blktap;
>
> - use LVM with blkback;
>
> - use upstream qemu with linux aio as device model and/or block backend.

Alex's real deployments use blkback on raw block devices so I think he
is ok.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 7:38 AM
Post #13 of 22 (1379 views)
Permalink
--On 27 October 2011 15:19:16 +0100 Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:

>> There seems to be some doubt (see Ian's message) about whether this
>> changes the backend driver that is used. The final deployment application
>> is tap:aio with a block device, so that's why we're doing this.
>
> If you are using XL, no matter if you specify tap:aio or file:, you are
> going to get qemu as disk backend if you are missing blktap.
> There is nothing wrong with that, except that qemu in 4.1 doesn't
> support linux aio so the performances are not very good. I am not sure
> which one is better: blkback on a loop device or qemu without linux aio,
> they are both rather slow.

I'm not sure I understand that. blkback on a loop device implies we
can just use blkback on a real device. We are using a real device
anyway in production (the file is just for testing).

So, should we be using tap:aio:/dev/... for a block device for
speed?

> In order to make it fast you can:
>
> - use a dom0 kernel that provides blktap;
>
> - use LVM with blkback;
>
> - use upstream qemu with linux aio as device model and/or block backend.

I /think/ you mean only if we are using a file, so that shouldn't
be relevant. Correct?

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
stefano.stabellini at eu
Oct 27, 2011, 7:51 AM
Post #14 of 22 (1389 views)
Permalink
On Thu, 27 Oct 2011, Alex Bligh wrote:
> --On 27 October 2011 15:19:16 +0100 Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>
> >> There seems to be some doubt (see Ian's message) about whether this
> >> changes the backend driver that is used. The final deployment application
> >> is tap:aio with a block device, so that's why we're doing this.
> >
> > If you are using XL, no matter if you specify tap:aio or file:, you are
> > going to get qemu as disk backend if you are missing blktap.
> > There is nothing wrong with that, except that qemu in 4.1 doesn't
> > support linux aio so the performances are not very good. I am not sure
> > which one is better: blkback on a loop device or qemu without linux aio,
> > they are both rather slow.
>
> I'm not sure I understand that. blkback on a loop device implies we
> can just use blkback on a real device. We are using a real device
> anyway in production (the file is just for testing).

Ahh, sorry, I missed that you are using real devices in production.


> So, should we be using tap:aio:/dev/... for a block device for
> speed?

You should use phy:/dev/whatever.
At this point I suggest you setup loop devices for testing to be
coherent.


> > In order to make it fast you can:
> >
> > - use a dom0 kernel that provides blktap;
> >
> > - use LVM with blkback;
> >
> > - use upstream qemu with linux aio as device model and/or block backend.
>
> I /think/ you mean only if we are using a file, so that shouldn't
> be relevant. Correct?

Correct.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 7:55 AM
Post #15 of 22 (1393 views)
Permalink
On Thu, 2011-10-27 at 15:38 +0100, Alex Bligh wrote:
>
> --On 27 October 2011 15:19:16 +0100 Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>
> >> There seems to be some doubt (see Ian's message) about whether this
> >> changes the backend driver that is used. The final deployment application
> >> is tap:aio with a block device, so that's why we're doing this.
> >
> > If you are using XL, no matter if you specify tap:aio or file:, you are
> > going to get qemu as disk backend if you are missing blktap.
> > There is nothing wrong with that, except that qemu in 4.1 doesn't
> > support linux aio so the performances are not very good. I am not sure
> > which one is better: blkback on a loop device or qemu without linux aio,
> > they are both rather slow.
>
> I'm not sure I understand that. blkback on a loop device implies we
> can just use blkback on a real device. We are using a real device
> anyway in production (the file is just for testing).
>
> So, should we be using tap:aio:/dev/... for a block device for
> speed?

tap:aio does not have a speed advantage -- in fact quite the opposite.
tap: gives you the flexibility to use non-raw block devices and
structured disk image types but is nothing like as fast as using a raw
block device with blkback.

The aio: part is an internal implementation detail of the tap: stuff
which should never have been exposed to the user.

Ian.

>
> > In order to make it fast you can:
> >
> > - use a dom0 kernel that provides blktap;
> >
> > - use LVM with blkback;
> >
> > - use upstream qemu with linux aio as device model and/or block backend.
>
> I /think/ you mean only if we are using a file, so that shouldn't
> be relevant. Correct?
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 7:57 AM
Post #16 of 22 (1382 views)
Permalink
--On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote:

> I think I've found one reason why we can't get PV block drivers on HVM
> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
> domU kernel.
>
> We are using a line like:
>
> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
>
> We are using effectively a standard 3.0 kernel. Config options including
> the word XEN are below.
>
> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> start in Xen3.

The magic incantation to fix this here was:
modprobe xen_gntdev
which for some reason is not modprobed in the startup script (Ubuntu
one anyway).

We now see pv block devices on hvm.

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 8:01 AM
Post #17 of 22 (1380 views)
Permalink
Stefano,

--On 27 October 2011 15:51:54 +0100 Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:

>> So, should we be using tap:aio:/dev/... for a block device for
>> speed?
>
> You should use phy:/dev/whatever.
> At this point I suggest you setup loop devices for testing to be
> coherent.

Thanks. As per my last message, it looks like we have discovered
the fundamental cause of the PV drivers not appearing (at least
for xenlinux domUs).

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
stefan.bader at canonical
Oct 27, 2011, 8:09 AM
Post #18 of 22 (1362 views)
Permalink
On 27.10.2011 16:57, Alex Bligh wrote:
>
>
> --On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote:
>
>> I think I've found one reason why we can't get PV block drivers on HVM
>> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
>> domU kernel.
>>
>> We are using a line like:
>>
>> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
>>
>> We are using effectively a standard 3.0 kernel. Config options including
>> the word XEN are below.
>>
>> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
>> start in Xen3.
>
> The magic incantation to fix this here was:
> modprobe xen_gntdev
> which for some reason is not modprobed in the startup script (Ubuntu
> one anyway).
>
> We now see pv block devices on hvm.
>
Hm, you mean in dom0? Strangely I do not have this loaded and still get pv disks
on hvm...

-Stefan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
Ian.Campbell at citrix
Oct 27, 2011, 8:13 AM
Post #19 of 22 (1362 views)
Permalink
On Thu, 2011-10-27 at 15:57 +0100, Alex Bligh wrote:
>
> --On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote:
>
> > I think I've found one reason why we can't get PV block drivers on HVM
> > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux
> > domU kernel.
> >
> > We are using a line like:
> >
> > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]
> >
> > We are using effectively a standard 3.0 kernel. Config options including
> > the word XEN are below.
> >
> > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does
> > start in Xen3.
>
> The magic incantation to fix this here was:
> modprobe xen_gntdev
> which for some reason is not modprobed in the startup script (Ubuntu
> one anyway).

I think 3.1 has patches to fix this in general (although I'm not sure
about this specific case).

> We now see pv block devices on hvm.

If you are using gntdev then are you running a qemu to provide the
backend?

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
konrad.wilk at oracle
Oct 27, 2011, 8:16 AM
Post #20 of 22 (1363 views)
Permalink
On Thu, Oct 27, 2011 at 04:01:13PM +0100, Alex Bligh wrote:
> Stefano,
>
> --On 27 October 2011 15:51:54 +0100 Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>
> >>So, should we be using tap:aio:/dev/... for a block device for
> >>speed?
> >
> >You should use phy:/dev/whatever.
> >At this point I suggest you setup loop devices for testing to be
> >coherent.

Or you can use Logical Volumes and dump your .img files in them and do:

dd if=/mnt/ViP_customer.img of=/dev/vg_guest/VIP_customer bs=1MB

and in your configuration file:
disk = ['phy:/dev/vg_guest/VIP_customer,xvda,w']

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 9:02 AM
Post #21 of 22 (1363 views)
Permalink
--On 27 October 2011 17:09:16 +0200 Stefan Bader
<stefan.bader@canonical.com> wrote:

>> We now see pv block devices on hvm.
>>
> Hm, you mean in dom0? Strangely I do not have this loaded and still get
> pv disks on hvm...

I mean that without doing
modprobe xen_gntdev
on dom0, I do not see pv block devices on hvm in a domU.

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Re: Re: PV drivers on HVM using Xen 4.1.1 [ In reply to ]
alex at alex
Oct 27, 2011, 9:04 AM
Post #22 of 22 (1365 views)
Permalink
Ian,

--On 27 October 2011 16:13:20 +0100 Ian Campbell <Ian.Campbell@citrix.com>
wrote:

> If you are using gntdev then are you running a qemu to provide the
> backend?

We are using tap:aio:/path/to/file which I think we have established
would fall back to qemu in the absence of blktap, as blkback only
works with block devices.

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal