Mailing List Archive

I'll answer myself.

Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
For some unexplained reasons, it calls O_DIRECT flag still:

strace ./td-util create vhd 2048 /path/to/file
open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
EINVAL (Invalid argument)
exit_group(22) = ?
+++ exited with 22 +++

Checking other files with O_WRONLY flag doesn't bring any ideas as none
of them use "O_DIRECT".

lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT, 0644);
tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);


Any ideas how to fix FileSR for ZFS on Linux?

Seems like I'm left with iScsi approach only.

Thanks,
Dawid

On 30/12/14 18:11, Dawid Kowalski wrote:
> Hi Guys,
>
> Just wanted to report that file based Storage Repository won't work on
> Linux systems as td-util is required O_DIRECT when creating image file
> not supported on ZFS (Linux).
>
> When creating vm, i.e. vm-import it calls td-util similar to below:
> td-util create vhd 2048 /path/to/file
>
> Strace shows:
> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
> EINVAL (Invalid argument)
>
> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty
> hack than proper solution of the problem.
>
> Regards,
> Dawid


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

> On 30 Dec 2014, at 17:24, Dawid Kowalski <dkadds2@gmail.com> wrote:
>
> I'll answer myself.
>
> Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
> For some unexplained reasons, it calls O_DIRECT flag still:

Hm. As an experiment you could try an LD_PRELOAD wrapper like this:

http://www.mcgill.org.za/stuff/software/nosync

Before using anything like this for data you care about, we need to think it through a little to make sure it’s safe. Perhaps Linux zfs support for iSCSI is safer? (Sorry to flip-flop on this, I’m not very familiar with ZFS)

Cheers,
Dave

>
> strace ./td-util create vhd 2048 /path/to/file
> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1 EINVAL (Invalid argument)
> exit_group(22) = ?
> +++ exited with 22 +++
>
> Checking other files with O_WRONLY flag doesn't bring any ideas as none of them use "O_DIRECT".
>
> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
> lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT, 0644);
> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>
>
> Any ideas how to fix FileSR for ZFS on Linux?
>
> Seems like I'm left with iScsi approach only.
>
> Thanks,
> Dawid
>
> On 30/12/14 18:11, Dawid Kowalski wrote:
>> Hi Guys,
>>
>> Just wanted to report that file based Storage Repository won't work on Linux systems as td-util is required O_DIRECT when creating image file not supported on ZFS (Linux).
>>
>> When creating vm, i.e. vm-import it calls td-util similar to below:
>> td-util create vhd 2048 /path/to/file
>>
>> Strace shows:
>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1 EINVAL (Invalid argument)
>>
>> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty hack than proper solution of the problem.
>>
>> Regards,
>> Dawid
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Just as a form of an update. I'm stuck trying to get NFS working as no
other way to use ZFS in background seems to be available for me. iSCSI
share option for ZFS is not available on Linux. Should I try to enable
it I'd need to add third-party daemon.

I'm looking forward for your help in NFS context.

In regards to nosync - I'm happy to give it a try as it's just a test
system now but I'd prefer to find working solution.

I've spent couple of days/evenings already fighting to get Xen working
on Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
complicated.

I'll still keep trying...

Thanks,
Dawid


On 30/12/14 18:40, Dave Scott wrote:
>> On 30 Dec 2014, at 17:24, Dawid Kowalski <dkadds2@gmail.com> wrote:
>>
>> I'll answer myself.
>>
>> Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
>> For some unexplained reasons, it calls O_DIRECT flag still:
> Hm. As an experiment you could try an LD_PRELOAD wrapper like this:
>
> http://www.mcgill.org.za/stuff/software/nosync
>
> Before using anything like this for data you care about, we need to think it through a little to make sure it’s safe. Perhaps Linux zfs support for iSCSI is safer? (Sorry to flip-flop on this, I’m not very familiar with ZFS)
>
> Cheers,
> Dave
>
>> strace ./td-util create vhd 2048 /path/to/file
>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1 EINVAL (Invalid argument)
>> exit_group(22) = ?
>> +++ exited with 22 +++
>>
>> Checking other files with O_WRONLY flag doesn't bring any ideas as none of them use "O_DIRECT".
>>
>> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
>> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
>> lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT, 0644);
>> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
>> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>>
>>
>> Any ideas how to fix FileSR for ZFS on Linux?
>>
>> Seems like I'm left with iScsi approach only.
>>
>> Thanks,
>> Dawid
>>
>> On 30/12/14 18:11, Dawid Kowalski wrote:
>>> Hi Guys,
>>>
>>> Just wanted to report that file based Storage Repository won't work on Linux systems as td-util is required O_DIRECT when creating image file not supported on ZFS (Linux).
>>>
>>> When creating vm, i.e. vm-import it calls td-util similar to below:
>>> td-util create vhd 2048 /path/to/file
>>>
>>> Strace shows:
>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1 EINVAL (Invalid argument)
>>>
>>> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty hack than proper solution of the problem.
>>>
>>> Regards,
>>> Dawid
>>
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Have you tried using ext4 on a ZVOL and putting your VHDs there?

Scroll down to the "Ext4 on a ZVOL" section:
https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/


On Tue, Dec 30, 2014 at 9:45 AM, Dawid Kowalski <dkadds2@gmail.com> wrote:

> Just as a form of an update. I'm stuck trying to get NFS working as no
> other way to use ZFS in background seems to be available for me. iSCSI
> share option for ZFS is not available on Linux. Should I try to enable it
> I'd need to add third-party daemon.
>
> I'm looking forward for your help in NFS context.
>
> In regards to nosync - I'm happy to give it a try as it's just a test
> system now but I'd prefer to find working solution.
>
> I've spent couple of days/evenings already fighting to get Xen working on
> Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
> complicated.
>
> I'll still keep trying...
>
> Thanks,
> Dawid
>
>
>
> On 30/12/14 18:40, Dave Scott wrote:
>
>> On 30 Dec 2014, at 17:24, Dawid Kowalski <dkadds2@gmail.com> wrote:
>>>
>>> I'll answer myself.
>>>
>>> Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
>>> For some unexplained reasons, it calls O_DIRECT flag still:
>>>
>> Hm. As an experiment you could try an LD_PRELOAD wrapper like this:
>>
>> http://www.mcgill.org.za/stuff/software/nosync
>>
>> Before using anything like this for data you care about, we need to think
>> it through a little to make sure it’s safe. Perhaps Linux zfs support for
>> iSCSI is safer? (Sorry to flip-flop on this, I’m not very familiar with ZFS)
>>
>> Cheers,
>> Dave
>>
>> strace ./td-util create vhd 2048 /path/to/file
>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>> EINVAL (Invalid argument)
>>> exit_group(22) = ?
>>> +++ exited with 22 +++
>>>
>>> Checking other files with O_WRONLY flag doesn't bring any ideas as none
>>> of them use "O_DIRECT".
>>>
>>> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
>>> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
>>> lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT, 0644);
>>> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
>>> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>>>
>>>
>>> Any ideas how to fix FileSR for ZFS on Linux?
>>>
>>> Seems like I'm left with iScsi approach only.
>>>
>>> Thanks,
>>> Dawid
>>>
>>> On 30/12/14 18:11, Dawid Kowalski wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> Just wanted to report that file based Storage Repository won't work on
>>>> Linux systems as td-util is required O_DIRECT when creating image file not
>>>> supported on ZFS (Linux).
>>>>
>>>> When creating vm, i.e. vm-import it calls td-util similar to below:
>>>> td-util create vhd 2048 /path/to/file
>>>>
>>>> Strace shows:
>>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>>> EINVAL (Invalid argument)
>>>>
>>>> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty
>>>> hack than proper solution of the problem.
>>>>
>>>> Regards,
>>>> Dawid
>>>>
>>>
>>> _______________________________________________
>>> Xen-api mailing list
>>> Xen-api@lists.xen.org
>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>>
>>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>

Hi Carlos,

Thanks for hint!
This one worked.

So what worked:
zpool -> zfs block device fomatted as ext4 and mounted as folder
storage repository created with ffs driver.
The other drivers, meaning :
- file, fileSR on mounted ext4 FS,
- ext/lvm pointed to /dev/zd0
- ffs directly on ZFS.

Failed.

Other last resort option was to create file on ZFS, losetup and create
VG on it.

FFS directly on ZFS failed most probably due to unsupported O_DIRECT
flag (err=22) and other failed most probably for similar reasons.

If anyone would have some questions, feel free to email me directly at
love-spam-remove-me-up-to-here-dkadds2+xen-api@gmail.com

Based on my experience, there seem to be a lot to be fixed on xen side
to be really user friendly/usable.

Based on ZFS list of features there should be direct connector from Xen
side.

Regards,
Dawid

On 30/12/14 20:04, Carlos Reategui wrote:
> Have you tried using ext4 on a ZVOL and putting your VHDs there?
>
> Scroll down to the "Ext4 on a ZVOL" section:
> https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/
>
>
> On Tue, Dec 30, 2014 at 9:45 AM, Dawid Kowalski <dkadds2@gmail.com
> <mailto:dkadds2@gmail.com>> wrote:
>
> Just as a form of an update. I'm stuck trying to get NFS working
> as no other way to use ZFS in background seems to be available for
> me. iSCSI share option for ZFS is not available on Linux. Should I
> try to enable it I'd need to add third-party daemon.
>
> I'm looking forward for your help in NFS context.
>
> In regards to nosync - I'm happy to give it a try as it's just a
> test system now but I'd prefer to find working solution.
>
> I've spent couple of days/evenings already fighting to get Xen
> working on Ubuntu with ZFS and I'm close to give up and try KVM.
> It's way to complicated.
>
> I'll still keep trying...
>
> Thanks,
> Dawid
>
>
>
> On 30/12/14 18:40, Dave Scott wrote:
>
> On 30 Dec 2014, at 17:24, Dawid Kowalski
> <dkadds2@gmail.com <mailto:dkadds2@gmail.com>> wrote:
>
> I'll answer myself.
>
> Modifying td.c line 258 to skip O_DIRECT and recompiling
> doesn't help.
> For some unexplained reasons, it calls O_DIRECT flag still:
>
> Hm. As an experiment you could try an LD_PRELOAD wrapper like
> this:
>
> http://www.mcgill.org.za/stuff/software/nosync
>
> Before using anything like this for data you care about, we
> need to think it through a little to make sure it’s safe.
> Perhaps Linux zfs support for iSCSI is safer? (Sorry to
> flip-flop on this, I’m not very familiar with ZFS)
>
> Cheers,
> Dave
>
> strace ./td-util create vhd 2048 /path/to/file
> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT,
> 0644) = -1 EINVAL (Invalid argument)
> exit_group(22) = ?
> +++ exited with 22 +++
>
> Checking other files with O_WRONLY flag doesn't bring any
> ideas as none of them use "O_DIRECT".
>
> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT |
> O_EXCL, 0644);
> lock.c: fd = open(lockfn_flink, O_WRONLY |
> O_CREAT, 0644);
> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT |
> O_NONBLOCK, 0666);
> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>
>
> Any ideas how to fix FileSR for ZFS on Linux?
>
> Seems like I'm left with iScsi approach only.
>
> Thanks,
> Dawid
>
> On 30/12/14 18:11, Dawid Kowalski wrote:
>
> Hi Guys,
>
> Just wanted to report that file based Storage
> Repository won't work on Linux systems as td-util is
> required O_DIRECT when creating image file not
> supported on ZFS (Linux).
>
> When creating vm, i.e. vm-import it calls td-util
> similar to below:
> td-util create vhd 2048 /path/to/file
>
> Strace shows:
> open("/path/to/file",
> O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1 EINVAL
> (Invalid argument)
>
> I'll try to patch td.c to skip O_DIRECT flag but this
> is rather dirty hack than proper solution of the problem.
>
> Regards,
> Dawid
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org <mailto:Xen-api@lists.xen.org>
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org <mailto:Xen-api@lists.xen.org>
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
>

Glad you got it to work.

BTW DirectIO support in ZFS on Linux is on the roadmap for 0.6.4 (
https://github.com/zfsonlinux/zfs/issues/224). However don't know when
that is scheduled for.

On Tue, Dec 30, 2014 at 4:04 PM, Dawid Kowalski <dkadds2@gmail.com> wrote:

> Hi Carlos,
>
> Thanks for hint!
> This one worked.
>
> So what worked:
> zpool -> zfs block device fomatted as ext4 and mounted as folder
> storage repository created with ffs driver.
> The other drivers, meaning :
> - file, fileSR on mounted ext4 FS,
> - ext/lvm pointed to /dev/zd0
> - ffs directly on ZFS.
>
> Failed.
>
> Other last resort option was to create file on ZFS, losetup and create VG
> on it.
>
> FFS directly on ZFS failed most probably due to unsupported O_DIRECT flag
> (err=22) and other failed most probably for similar reasons.
>
> If anyone would have some questions, feel free to email me directly at
> love-spam-remove-me-up-to-here-dkadds2+xen-api@gmail.com
>
> Based on my experience, there seem to be a lot to be fixed on xen side to
> be really user friendly/usable.
>
> Based on ZFS list of features there should be direct connector from Xen
> side.
>
> Regards,
> Dawid
>
>
> On 30/12/14 20:04, Carlos Reategui wrote:
>
> Have you tried using ext4 on a ZVOL and putting your VHDs there?
>
> Scroll down to the "Ext4 on a ZVOL" section:
> https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/
>
>
> On Tue, Dec 30, 2014 at 9:45 AM, Dawid Kowalski <dkadds2@gmail.com> wrote:
>
>> Just as a form of an update. I'm stuck trying to get NFS working as no
>> other way to use ZFS in background seems to be available for me. iSCSI
>> share option for ZFS is not available on Linux. Should I try to enable it
>> I'd need to add third-party daemon.
>>
>> I'm looking forward for your help in NFS context.
>>
>> In regards to nosync - I'm happy to give it a try as it's just a test
>> system now but I'd prefer to find working solution.
>>
>> I've spent couple of days/evenings already fighting to get Xen working on
>> Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
>> complicated.
>>
>> I'll still keep trying...
>>
>> Thanks,
>> Dawid
>>
>>
>>
>> On 30/12/14 18:40, Dave Scott wrote:
>>
>>> On 30 Dec 2014, at 17:24, Dawid Kowalski <dkadds2@gmail.com> wrote:
>>>>
>>>> I'll answer myself.
>>>>
>>>> Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
>>>> For some unexplained reasons, it calls O_DIRECT flag still:
>>>>
>>> Hm. As an experiment you could try an LD_PRELOAD wrapper like this:
>>>
>>> http://www.mcgill.org.za/stuff/software/nosync
>>>
>>> Before using anything like this for data you care about, we need to
>>> think it through a little to make sure it’s safe. Perhaps Linux zfs support
>>> for iSCSI is safer? (Sorry to flip-flop on this, I’m not very familiar with
>>> ZFS)
>>>
>>> Cheers,
>>> Dave
>>>
>>> strace ./td-util create vhd 2048 /path/to/file
>>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>>> EINVAL (Invalid argument)
>>>> exit_group(22) = ?
>>>> +++ exited with 22 +++
>>>>
>>>> Checking other files with O_WRONLY flag doesn't bring any ideas as none
>>>> of them use "O_DIRECT".
>>>>
>>>> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
>>>> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
>>>> lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT,
>>>> 0644);
>>>> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
>>>> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>>>>
>>>>
>>>> Any ideas how to fix FileSR for ZFS on Linux?
>>>>
>>>> Seems like I'm left with iScsi approach only.
>>>>
>>>> Thanks,
>>>> Dawid
>>>>
>>>> On 30/12/14 18:11, Dawid Kowalski wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> Just wanted to report that file based Storage Repository won't work on
>>>>> Linux systems as td-util is required O_DIRECT when creating image file not
>>>>> supported on ZFS (Linux).
>>>>>
>>>>> When creating vm, i.e. vm-import it calls td-util similar to below:
>>>>> td-util create vhd 2048 /path/to/file
>>>>>
>>>>> Strace shows:
>>>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>>>> EINVAL (Invalid argument)
>>>>>
>>>>> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty
>>>>> hack than proper solution of the problem.
>>>>>
>>>>> Regards,
>>>>> Dawid
>>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-api mailing list
>>>> Xen-api@lists.xen.org
>>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>>>
>>>
>>
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>
>
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
>

Hi Dawid,

Which sm or XS version are you using?
If you are using 6.2 or earlier, yes the fs need to support direct IO.

In the master branch and xs64bit there is code to disable o_direct.

Those changes go hand in hand with corresponding blktap changes.

Cheers,
Germano

On 12/30/2014 05:45 PM, Dawid Kowalski wrote:
> Just as a form of an update. I'm stuck trying to get NFS working as no
> other way to use ZFS in background seems to be available for me. iSCSI
> share option for ZFS is not available on Linux. Should I try to enable
> it I'd need to add third-party daemon.
>
> I'm looking forward for your help in NFS context.
>
> In regards to nosync - I'm happy to give it a try as it's just a test
> system now but I'd prefer to find working solution.
>
> I've spent couple of days/evenings already fighting to get Xen working
> on Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
> complicated.
>
> I'll still keep trying...
>


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Thanks Carlos.
Interesting info on ZFS O_DIRECT :)

Regards,
Dawid

2014-12-31 2:03 GMT+01:00 Carlos Reategui <carlos@reategui.com>:

> Glad you got it to work.
>
> BTW DirectIO support in ZFS on Linux is on the roadmap for 0.6.4 (
> https://github.com/zfsonlinux/zfs/issues/224). However don't know when
> that is scheduled for.
>
> On Tue, Dec 30, 2014 at 4:04 PM, Dawid Kowalski <dkadds2@gmail.com> wrote:
>
>> Hi Carlos,
>>
>> Thanks for hint!
>> This one worked.
>>
>> So what worked:
>> zpool -> zfs block device fomatted as ext4 and mounted as folder
>> storage repository created with ffs driver.
>> The other drivers, meaning :
>> - file, fileSR on mounted ext4 FS,
>> - ext/lvm pointed to /dev/zd0
>> - ffs directly on ZFS.
>>
>> Failed.
>>
>> Other last resort option was to create file on ZFS, losetup and create VG
>> on it.
>>
>> FFS directly on ZFS failed most probably due to unsupported O_DIRECT flag
>> (err=22) and other failed most probably for similar reasons.
>>
>> If anyone would have some questions, feel free to email me directly at
>> love-spam-remove-me-up-to-here-dkadds2+xen-api@gmail.com
>>
>> Based on my experience, there seem to be a lot to be fixed on xen side to
>> be really user friendly/usable.
>>
>> Based on ZFS list of features there should be direct connector from Xen
>> side.
>>
>> Regards,
>> Dawid
>>
>>
>> On 30/12/14 20:04, Carlos Reategui wrote:
>>
>> Have you tried using ext4 on a ZVOL and putting your VHDs there?
>>
>> Scroll down to the "Ext4 on a ZVOL" section:
>> https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/
>>
>>
>> On Tue, Dec 30, 2014 at 9:45 AM, Dawid Kowalski <dkadds2@gmail.com>
>> wrote:
>>
>>> Just as a form of an update. I'm stuck trying to get NFS working as no
>>> other way to use ZFS in background seems to be available for me. iSCSI
>>> share option for ZFS is not available on Linux. Should I try to enable it
>>> I'd need to add third-party daemon.
>>>
>>> I'm looking forward for your help in NFS context.
>>>
>>> In regards to nosync - I'm happy to give it a try as it's just a test
>>> system now but I'd prefer to find working solution.
>>>
>>> I've spent couple of days/evenings already fighting to get Xen working
>>> on Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
>>> complicated.
>>>
>>> I'll still keep trying...
>>>
>>> Thanks,
>>> Dawid
>>>
>>>
>>>
>>> On 30/12/14 18:40, Dave Scott wrote:
>>>
>>>> On 30 Dec 2014, at 17:24, Dawid Kowalski <dkadds2@gmail.com> wrote:
>>>>>
>>>>> I'll answer myself.
>>>>>
>>>>> Modifying td.c line 258 to skip O_DIRECT and recompiling doesn't help.
>>>>> For some unexplained reasons, it calls O_DIRECT flag still:
>>>>>
>>>> Hm. As an experiment you could try an LD_PRELOAD wrapper like this:
>>>>
>>>> http://www.mcgill.org.za/stuff/software/nosync
>>>>
>>>> Before using anything like this for data you care about, we need to
>>>> think it through a little to make sure it’s safe. Perhaps Linux zfs support
>>>> for iSCSI is safer? (Sorry to flip-flop on this, I’m not very familiar with
>>>> ZFS)
>>>>
>>>> Cheers,
>>>> Dave
>>>>
>>>> strace ./td-util create vhd 2048 /path/to/file
>>>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>>>> EINVAL (Invalid argument)
>>>>> exit_group(22) = ?
>>>>> +++ exited with 22 +++
>>>>>
>>>>> Checking other files with O_WRONLY flag doesn't bring any ideas as
>>>>> none of them use "O_DIRECT".
>>>>>
>>>>> lock.c: fd = open(buf, O_WRONLY | O_CREAT, 0644);
>>>>> lock.c: fd = open(lockfn, O_WRONLY | O_CREAT | O_EXCL, 0644);
>>>>> lock.c: fd = open(lockfn_flink, O_WRONLY | O_CREAT,
>>>>> 0644);
>>>>> tapdisk-vbd.c: fd = open(fn, O_WRONLY | O_CREAT | O_NONBLOCK, 0666);
>>>>> td.c: fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
>>>>>
>>>>>
>>>>> Any ideas how to fix FileSR for ZFS on Linux?
>>>>>
>>>>> Seems like I'm left with iScsi approach only.
>>>>>
>>>>> Thanks,
>>>>> Dawid
>>>>>
>>>>> On 30/12/14 18:11, Dawid Kowalski wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> Just wanted to report that file based Storage Repository won't work
>>>>>> on Linux systems as td-util is required O_DIRECT when creating image file
>>>>>> not supported on ZFS (Linux).
>>>>>>
>>>>>> When creating vm, i.e. vm-import it calls td-util similar to below:
>>>>>> td-util create vhd 2048 /path/to/file
>>>>>>
>>>>>> Strace shows:
>>>>>> open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0644) = -1
>>>>>> EINVAL (Invalid argument)
>>>>>>
>>>>>> I'll try to patch td.c to skip O_DIRECT flag but this is rather dirty
>>>>>> hack than proper solution of the problem.
>>>>>>
>>>>>> Regards,
>>>>>> Dawid
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-api mailing list
>>>>> Xen-api@lists.xen.org
>>>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Xen-api mailing list
>>> Xen-api@lists.xen.org
>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>>
>>
>>
>>
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>>
>>
>

Thanks Germano,

I'm running latest master branch taken from here:

git clone git://github.com/xenserver/buildroot.git

as per:
https://github.com/xenserver/buildroot

If you could recommend me any good working branch I might be keen to try it.

btw. slightly off topic...
I've to admit that during last days I've run into so many issues that
almost gave up using Xen. But on the other hand I'm not sure if KVM is any
better.
For contrary, VirtualBox even if is headed towards desktop user, for couple
of VMs as I need at the moment could be a better solution for me... It's
sad to get to such conclusions, but given the time I've spent
troubleshooting and running from issue to issue... It makes me thinking Xen
is more in beta stage than real production one.

Regards,
Dawid


2015-01-06 13:13 GMT+01:00 Germano Percossi <germano.percossi@citrix.com>:

> Hi Dawid,
>
> Which sm or XS version are you using?
> If you are using 6.2 or earlier, yes the fs need to support direct IO.
>
> In the master branch and xs64bit there is code to disable o_direct.
>
> Those changes go hand in hand with corresponding blktap changes.
>
> Cheers,
> Germano
>
> On 12/30/2014 05:45 PM, Dawid Kowalski wrote:
>
>> Just as a form of an update. I'm stuck trying to get NFS working as no
>> other way to use ZFS in background seems to be available for me. iSCSI
>> share option for ZFS is not available on Linux. Should I try to enable
>> it I'd need to add third-party daemon.
>>
>> I'm looking forward for your help in NFS context.
>>
>> In regards to nosync - I'm happy to give it a try as it's just a test
>> system now but I'd prefer to find working solution.
>>
>> I've spent couple of days/evenings already fighting to get Xen working
>> on Ubuntu with ZFS and I'm close to give up and try KVM. It's way to
>> complicated.
>>
>> I'll still keep trying...
>>
>>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>

Hi Dawid,

For what concerns your build I do not know how it maps to storage
manager and blktap versions.
If in /opt/xensource/sm/blktap2.py there is a line like

CONF_KEY_O_DIRECT = "o_direct"

then you can disable o_direct.

About your frustration I can understand but it is a long shot
to say xenserver is in beta stage, given there are many huge
business running on top of it with almost no downtime, reliability
and speed.
Surely it is not bug free and surely it is not able to cope with
any kind of user scenario.
The same, though, is true for kvm and virtualbox (just to name the
ones you named).
Maybe your use case is much closer to virtualbox's.

The ML is here to help, as much as possible.
If not and other tools are more suitable for your needs,
that's fine :)

Cheers,
Germano

On 01/06/2015 06:02 PM, Dawid Kowalski wrote:
> Thanks Germano,
>
> I'm running latest master branch taken from here:
>
> |git clone git://github.com/xenserver/buildroot.git <http://github.com/xenserver/buildroot.git>|
>
> as per:
> https://github.com/xenserver/buildroot
>
> If you could recommend me any good working branch I might be keen to try it.
>
> btw. slightly off topic...
> I've to admit that during last days I've run into so many issues that
> almost gave up using Xen. But on the other hand I'm not sure if KVM is
> any better.
> For contrary, VirtualBox even if is headed towards desktop user, for
> couple of VMs as I need at the moment could be a better solution for
> me... It's sad to get to such conclusions, but given the time I've spent
> troubleshooting and running from issue to issue... It makes me thinking
> Xen is more in beta stage than real production one.
>
> Regards,
> Dawid


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Germano,

Many helps for your tips.
Looks like I'm running even if latest master branch, some kind of old
version as all I have is:
grep CONF_KEY_ /usr/lib/xapi/sm/blktap2.py
CONF_KEY_ALLOW_CACHING = "vdi_allow_caching"
CONF_KEY_MODE_ON_BOOT = "vdi_on_boot"
CONF_KEY_CACHE_SR = "local_cache_sr"

If you're anyhow familiar which Xenproject even non GA version it the right
one to go, that would be great.
All I need is to put it on top of base Ubuntu 14.04 as can't go with
packaged XenServer.iso due to some requirements on my side (full hdd
necryption).

Certainly I don't question power behind Xen, as this is one of the reasons
why I turned into this direction. Personally I do know large companies
using Xen/CitrixXen implementation for long time already.

It's only that in my case I've run into so many issues, that it brought me
to thoughts as expressed earlier. Regardless of issues I've run into, using
different workarounds I'm heading to my target.

What I really like about Xen is the PVM.

All help from you and others is more than appreciated. Thank you!
Dawid

2015-01-06 20:25 GMT+01:00 Germano Percossi <germano.percossi@citrix.com>:

> Hi Dawid,
>
> For what concerns your build I do not know how it maps to storage
> manager and blktap versions.
> If in /opt/xensource/sm/blktap2.py there is a line like
>
> CONF_KEY_O_DIRECT = "o_direct"
>
> then you can disable o_direct.
>
> About your frustration I can understand but it is a long shot
> to say xenserver is in beta stage, given there are many huge
> business running on top of it with almost no downtime, reliability
> and speed.
> Surely it is not bug free and surely it is not able to cope with
> any kind of user scenario.
> The same, though, is true for kvm and virtualbox (just to name the
> ones you named).
> Maybe your use case is much closer to virtualbox's.
>
> The ML is here to help, as much as possible.
> If not and other tools are more suitable for your needs,
> that's fine :)
>
> Cheers,
> Germano
>
> On 01/06/2015 06:02 PM, Dawid Kowalski wrote:
>
>> Thanks Germano,
>>
>> I'm running latest master branch taken from here:
>>
>> |git clone git://github.com/xenserver/buildroot.git <
>> http://github.com/xenserver/buildroot.git>|
>>
>> as per:
>> https://github.com/xenserver/buildroot
>>
>> If you could recommend me any good working branch I might be keen to try
>> it.
>>
>> btw. slightly off topic...
>> I've to admit that during last days I've run into so many issues that
>> almost gave up using Xen. But on the other hand I'm not sure if KVM is
>> any better.
>> For contrary, VirtualBox even if is headed towards desktop user, for
>> couple of VMs as I need at the moment could be a better solution for
>> me... It's sad to get to such conclusions, but given the time I've spent
>> troubleshooting and running from issue to issue... It makes me thinking
>> Xen is more in beta stage than real production one.
>>
>> Regards,
>> Dawid
>>
>
>