Mailing List Archive

Bad performance with Xen
Hello.


We are testing low performance in IO with the next command in Debian Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)

       time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null


In all our Dom0s - DomUs  we are getting around 20 seconds.

In the same physical machines booting with Debian without Xen, we get 5-7 seconds

In some KVM VMs in other server we are geting almost the same as physical.

(all in local Disks. XFS filesystems. Images of DomUs in raw format)


I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.


Where could be the problem?

I think of is not normal this difference between DomUs and physical machine.


Every pointer will be welcomed.


Best regards,

Agustín
Re: Bad performance with Xen [ In reply to ]
Hi Agustin,

I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM, and
here is the result I have:

```
# time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls
-ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null

real 0m2,741s
user 0m2,248s
sys 0m0,574s
```

My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a small
Dell T30 machine, VM storage on local HDD. I did the test 3 times, and I
have always results between 2,6 and 2,8 secs.

Regards,

Olivier.

Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :

> Hello.
>
>
> We are testing low performance in IO with the next command in Debian
> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>
> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ];
> then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>
>
> In all our Dom0s - DomUs we are getting around 20 seconds.
>
> In the same physical machines booting with Debian without Xen, we get 5-7
> seconds
>
> In some KVM VMs in other server we are geting almost the same as physical.
>
> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>
>
> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>
>
> Where could be the problem?
>
> I think of is not normal this difference between DomUs and physical
> machine.
>
>
> Every pointer will be welcomed.
>
>
> Best regards,
>
> Agustín
>
>
>
>
>
>
Re: Bad performance with Xen [ In reply to ]
Hi Oliver.

I am testing a bit more. In seconds, the results of the command is:
Debian Buster PV -> 18'
Debian Buster HVM -> 8'
Debian Buster PVHVM -> 8'
Debian Buster PVH -> 8'


xl info
release                : 4.19.0-8-amd64
version                : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
machine                : x86_64
nr_cpus                : 48
max_cpu_id             : 47
nr_nodes               : 2
cores_per_socket       : 12
threads_per_core       : 2
cpu_mhz                : 2197.458
hw_caps                : bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 261890
free_memory            : 255453
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 11
xen_extra              : .4-pre
xen_version            : 4.11.4-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=2048M,max:4065M
cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by          : pkg-xen-devel
cc_compile_domain      : lists.alioth.debian.org
cc_compile_date        : Wed Jan  8 20:16:51 UTC 2020
build_id               : b6822aa1d8f867753b92985e5cb0e806e520a08c
xend_config_format     : 4

Oliver, I got > double values than you. Where is the problem?

Regards,

Agustín



El 2/5/20 a las 19:56, Olivier Lambert escribió:
> Hi Agustin,
>
> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM, and here is the result I have:
>
> ```
> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>
> real 0m2,741s
> user 0m2,248s
> sys 0m0,574s
> ```
>
> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a small Dell T30 machine, VM storage on local HDD. I did the test 3 times, and I have always results between 2,6 and 2,8 secs.
>
> Regards,
>
> Olivier.
>
> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a écrit :
>
> Hello.
>
>
> We are testing low performance in IO with the next command in Debian Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>
>         time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>
>
> In all our Dom0s - DomUs  we are getting around 20 seconds.
>
> In the same physical machines booting with Debian without Xen, we get 5-7 seconds
>
> In some KVM VMs in other server we are geting almost the same as physical.
>
> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>
>
> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>
>
> Where could be the problem?
>
> I think of is not normal this difference between DomUs and physical machine.
>
>
> Every pointer will be welcomed.
>
>
> Best regards,
>
> Agustín
>
>
>
>
>
Re: Bad performance with Xen [ In reply to ]
Hard to tell. Here is my xl info to compare:

host : xcp-ng-lab-3
release : 4.19.0+1
version : #1 SMP Thu Feb 13 17:34:28 CET 2020
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 3312.134
hw_caps :
bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps : pv hvm hvm_directio pv_directio hap shadow
iommu_hap_pt_share
total_memory : 32634
free_memory : 23619
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 13
xen_extra : .0-8.4.xcpng8.1
xen_version : 4.13.0-8.4.xcpng8.1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : 85e1424de2dd, pq f9dbf852550e
xen_commandline : watchdog ucode=scan dom0_max_vcpus=1-4
crashkernel=256M,below=4G console=vga vga=mode-0x0311
dom0_mem=8192M,max:8192M
cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
cc_compile_by : mockbuild
cc_compile_domain : [unknown]
cc_compile_date : Tue Apr 14 18:28:14 CEST 2020
build_id : 5ad6f12499d7f264544b64568b378260cd82a65f
xend_config_format : 4

I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I ran
the test on another server (building a VM just for you :p ) and here is the
result for a Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored on a NFS
share.

real 0m5,925s
user 0m3,769s
sys 0m2,321s

Still, far better than 20 seconds you have!

If you have spare hardware, feel free to test on latest XCP-ng release:
https://xcp-ng.org/docs/install.html

Let me know if you need further help :)


Best,

Olivier.


Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :

> Hi Oliver.
>
> I am testing a bit more. In seconds, the results of the command is:
> Debian Buster PV -> 18'
> Debian Buster HVM -> 8'
> Debian Buster PVHVM -> 8'
> Debian Buster PVH -> 8'
>
>
> xl info
> release : 4.19.0-8-amd64
> version : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
> machine : x86_64
> nr_cpus : 48
> max_cpu_id : 47
> nr_nodes : 2
> cores_per_socket : 12
> threads_per_core : 2
> cpu_mhz : 2197.458
> hw_caps :
> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
> virt_caps : hvm hvm_directio
> total_memory : 261890
> free_memory : 255453
> sharing_freed_memory : 0
> sharing_used_memory : 0
> outstanding_claims : 0
> free_cpus : 0
> xen_major : 4
> xen_minor : 11
> xen_extra : .4-pre
> xen_version : 4.11.4-pre
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset :
> xen_commandline : placeholder dom0_mem=2048M,max:4065M
> cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
> cc_compile_by : pkg-xen-devel
> cc_compile_domain : lists.alioth.debian.org
> cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
> build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
> xend_config_format : 4
>
> Oliver, I got > double values than you. Where is the problem?
>
> Regards,
>
> Agustín
>
>
>
> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>
> Hi Agustin,
>
> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM, and
> here is the result I have:
>
> ```
> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls
> -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>
> real 0m2,741s
> user 0m2,248s
> sys 0m0,574s
> ```
>
> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a small
> Dell T30 machine, VM storage on local HDD. I did the test 3 times, and I
> have always results between 2,6 and 2,8 secs.
>
> Regards,
>
> Olivier.
>
> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :
>
>> Hello.
>>
>>
>> We are testing low performance in IO with the next command in Debian
>> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>
>> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ];
>> then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>
>>
>> In all our Dom0s - DomUs we are getting around 20 seconds.
>>
>> In the same physical machines booting with Debian without Xen, we get 5-7
>> seconds
>>
>> In some KVM VMs in other server we are geting almost the same as physical.
>>
>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>
>>
>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>>
>>
>> Where could be the problem?
>>
>> I think of is not normal this difference between DomUs and physical
>> machine.
>>
>>
>> Every pointer will be welcomed.
>>
>>
>> Best regards,
>>
>> Agustín
>>
>>
>>
>>
>>
>>
Re: Bad performance with Xen [ In reply to ]
Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
so increasing RAM allocated to Dom0 might speed up the VMs.

2GB for dom0 is extremely low in my opinion especially when most of the
256GB of host RAM is going to waste.

dom0_mem=2048M,max:4065M

On Sun, May 3, 2020 at 10:41 AM Olivier Lambert <lambert.olivier@gmail.com>
wrote:

> Hard to tell. Here is my xl info to compare:
>
> host : xcp-ng-lab-3
> release : 4.19.0+1
> version : #1 SMP Thu Feb 13 17:34:28 CET 2020
> machine : x86_64
> nr_cpus : 4
> max_cpu_id : 3
> nr_nodes : 1
> cores_per_socket : 4
> threads_per_core : 1
> cpu_mhz : 3312.134
> hw_caps :
> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
> virt_caps : pv hvm hvm_directio pv_directio hap shadow
> iommu_hap_pt_share
> total_memory : 32634
> free_memory : 23619
> sharing_freed_memory : 0
> sharing_used_memory : 0
> outstanding_claims : 0
> free_cpus : 0
> xen_major : 4
> xen_minor : 13
> xen_extra : .0-8.4.xcpng8.1
> xen_version : 4.13.0-8.4.xcpng8.1
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset : 85e1424de2dd, pq f9dbf852550e
> xen_commandline : watchdog ucode=scan dom0_max_vcpus=1-4
> crashkernel=256M,below=4G console=vga vga=mode-0x0311
> dom0_mem=8192M,max:8192M
> cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
> cc_compile_by : mockbuild
> cc_compile_domain : [unknown]
> cc_compile_date : Tue Apr 14 18:28:14 CEST 2020
> build_id : 5ad6f12499d7f264544b64568b378260cd82a65f
> xend_config_format : 4
>
> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I ran
> the test on another server (building a VM just for you :p ) and here is the
> result for a Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored on a NFS
> share.
>
> real 0m5,925s
> user 0m3,769s
> sys 0m2,321s
>
> Still, far better than 20 seconds you have!
>
> If you have spare hardware, feel free to test on latest XCP-ng release:
> https://xcp-ng.org/docs/install.html
>
> Let me know if you need further help :)
>
>
> Best,
>
> Olivier.
>
>
> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :
>
>> Hi Oliver.
>>
>> I am testing a bit more. In seconds, the results of the command is:
>> Debian Buster PV -> 18'
>> Debian Buster HVM -> 8'
>> Debian Buster PVHVM -> 8'
>> Debian Buster PVH -> 8'
>>
>>
>> xl info
>> release : 4.19.0-8-amd64
>> version : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
>> machine : x86_64
>> nr_cpus : 48
>> max_cpu_id : 47
>> nr_nodes : 2
>> cores_per_socket : 12
>> threads_per_core : 2
>> cpu_mhz : 2197.458
>> hw_caps :
>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>> virt_caps : hvm hvm_directio
>> total_memory : 261890
>> free_memory : 255453
>> sharing_freed_memory : 0
>> sharing_used_memory : 0
>> outstanding_claims : 0
>> free_cpus : 0
>> xen_major : 4
>> xen_minor : 11
>> xen_extra : .4-pre
>> xen_version : 4.11.4-pre
>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>> hvm-3.0-x86_32p hvm-3.0-x86_64
>> xen_scheduler : credit
>> xen_pagesize : 4096
>> platform_params : virt_start=0xffff800000000000
>> xen_changeset :
>> xen_commandline : placeholder dom0_mem=2048M,max:4065M
>> cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
>> cc_compile_by : pkg-xen-devel
>> cc_compile_domain : lists.alioth.debian.org
>> cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
>> build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
>> xend_config_format : 4
>>
>> Oliver, I got > double values than you. Where is the problem?
>>
>> Regards,
>>
>> Agustín
>>
>>
>>
>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>
>> Hi Agustin,
>>
>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM, and
>> here is the result I have:
>>
>> ```
>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls
>> -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>
>> real 0m2,741s
>> user 0m2,248s
>> sys 0m0,574s
>> ```
>>
>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a
>> small Dell T30 machine, VM storage on local HDD. I did the test 3 times,
>> and I have always results between 2,6 and 2,8 secs.
>>
>> Regards,
>>
>> Olivier.
>>
>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :
>>
>>> Hello.
>>>
>>>
>>> We are testing low performance in IO with the next command in Debian
>>> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>>
>>> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ];
>>> then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>
>>>
>>> In all our Dom0s - DomUs we are getting around 20 seconds.
>>>
>>> In the same physical machines booting with Debian without Xen, we get
>>> 5-7 seconds
>>>
>>> In some KVM VMs in other server we are geting almost the same as
>>> physical.
>>>
>>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>>
>>>
>>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>>>
>>>
>>> Where could be the problem?
>>>
>>> I think of is not normal this difference between DomUs and physical
>>> machine.
>>>
>>>
>>> Every pointer will be welcomed.
>>>
>>>
>>> Best regards,
>>>
>>> Agustín
>>>
>>>
>>>
>>>
>>>
>>>
Re: Bad performance with Xen [ In reply to ]
Sorry. I booted with 8 GB for the Dom0 and all is the same.

I have seen one difference between the 2 xl info:

(AGUSTIN) virt_caps              : hvm hvm_directio

(OLIVIER) virt_caps              : pv hvm hvm_directio pv_directio hap shadow iommu_hap_pt_share


Could this be the problem?


Agustín


El 3/5/20 a las 18:50, Rob Townley escribió:
> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
> so increasing RAM allocated to Dom0 might speed up the VMs.
>
> 2GB for dom0 is extremely low in my opinion especially when most of the 256GB of host RAM is going to waste.
>
> dom0_mem=2048M,max:4065M
>
> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert <lambert.olivier@gmail.com <mailto:lambert.olivier@gmail.com>> wrote:
>
> Hard to tell. Here is my xl info to compare:
>
> host                   : xcp-ng-lab-3
> release                : 4.19.0+1
> version                : #1 SMP Thu Feb 13 17:34:28 CET 2020
> machine                : x86_64
> nr_cpus                : 4
> max_cpu_id             : 3
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 3312.134
> hw_caps                : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
> virt_caps              : pv hvm hvm_directio pv_directio hap shadow iommu_hap_pt_share
> total_memory           : 32634
> free_memory            : 23619
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 13
> xen_extra              : .0-8.4.xcpng8.1
> xen_version            : 4.13.0-8.4.xcpng8.1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : 85e1424de2dd, pq f9dbf852550e
> xen_commandline        : watchdog ucode=scan dom0_max_vcpus=1-4 crashkernel=256M,below=4G console=vga vga=mode-0x0311 dom0_mem=8192M,max:8192M
> cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
> cc_compile_by          : mockbuild
> cc_compile_domain      : [unknown]
> cc_compile_date        : Tue Apr 14 18:28:14 CEST 2020
> build_id               : 5ad6f12499d7f264544b64568b378260cd82a65f
> xend_config_format     : 4
>
> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I ran the test on another server (building a VM just for you :p ) and here is the result for a Xeon E5-2650L v2 @ 1.70GHz
> (slow!) and VM disk stored on a NFS share.
>
> real 0m5,925s
> user 0m3,769s
> sys 0m2,321s
>
> Still, far better than 20 seconds you have!
>
> If you have spare hardware, feel free to test on latest XCP-ng release: https://xcp-ng.org/docs/install.html
>
> Let me know if you need further help :)
>
>
> Best,
>
> Olivier.
>
>
> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a écrit :
>
> Hi Oliver.
>
> I am testing a bit more. In seconds, the results of the command is:
> Debian Buster PV -> 18'
> Debian Buster HVM -> 8'
> Debian Buster PVHVM -> 8'
> Debian Buster PVH -> 8'
>
>
> xl info
> release                : 4.19.0-8-amd64
> version                : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
> machine                : x86_64
> nr_cpus                : 48
> max_cpu_id             : 47
> nr_nodes               : 2
> cores_per_socket       : 12
> threads_per_core       : 2
> cpu_mhz                : 2197.458
> hw_caps                : bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
> virt_caps              : hvm hvm_directio
> total_memory           : 261890
> free_memory            : 255453
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 11
> xen_extra              : .4-pre
> xen_version            : 4.11.4-pre
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : placeholder dom0_mem=2048M,max:4065M
> cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
> cc_compile_by          : pkg-xen-devel
> cc_compile_domain      : lists.alioth.debian.org <http://lists.alioth.debian.org>
> cc_compile_date        : Wed Jan  8 20:16:51 UTC 2020
> build_id               : b6822aa1d8f867753b92985e5cb0e806e520a08c
> xend_config_format     : 4
>
> Oliver, I got > double values than you. Where is the problem?
>
> Regards,
>
> Agustín
>
>
>
> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>> Hi Agustin,
>>
>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM, and here is the result I have:
>>
>> ```
>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>
>> real 0m2,741s
>> user 0m2,248s
>> sys 0m0,574s
>> ```
>>
>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a small Dell T30 machine, VM storage on local HDD. I did the test 3 times, and I have always results between 2,6 and 2,8 secs.
>>
>> Regards,
>>
>> Olivier.
>>
>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a écrit :
>>
>> Hello.
>>
>>
>> We are testing low performance in IO with the next command in Debian Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>
>>         time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>
>>
>> In all our Dom0s - DomUs  we are getting around 20 seconds.
>>
>> In the same physical machines booting with Debian without Xen, we get 5-7 seconds
>>
>> In some KVM VMs in other server we are geting almost the same as physical.
>>
>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>
>>
>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>>
>>
>> Where could be the problem?
>>
>> I think of is not normal this difference between DomUs and physical machine.
>>
>>
>> Every pointer will be welcomed.
>>
>>
>> Best regards,
>>
>> Agustín
>>
>>
>>
>>
>>
Re: Bad performance with Xen [ In reply to ]
Hi guys. Maybe we are suffering some related issue. If not, feel free to
ignore this message.
I wrote on this list but none replyed:

"Fresh installed server with Debian Buster on top of nvme swRaid1 (mdadm)
Testing hdd write seed with dd (with convert=fdatasync option) gives me
the result of 330MB/s. Good.
Installed xen-system and xen-tools (with --no-recommends option in apt)
from official repository. Rebooted the system.
Re-tested hdd write seed with dd (with convert=fdatasync option) gives
me the result of 108MB/s. Not good at all.

Maybe the following is not related to the issue, but on dmesg there is a
line when I boot the system with Xen kernel:
...
[   14.214044] Performance Events: unsupported p6 CPU model 158 no PMU
driver, software events only.
...

Instead, when I boot the system without Xen kernel I have these lines in
dmesg:
...
[    0.517217] Performance Events: PEBS fmt3+, Skylake events, 32-deep
LBR, full-width counters, Intel PMU driver.
[    0.517356] ... version:                4
[    0.517444] ... bit width:              48
[    0.517444] ... generic registers:      4
[    0.517444] ... value mask:             0000ffffffffffff
[    0.517444] ... max period:             00007fffffffffff
[    0.517444] ... fixed-purpose events:   3
[    0.517444] ... event mask:             000000070000000f

"

Personally, I moved to KVM+libvirt nearly without rework.
I/O performance are great.
But I love XEN and I will be pleased to come back to it.
g


On 03/05/20 19:24, Agustin Lopez wrote:
>
>
> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>
> I have seen one difference between the 2 xl info:
>
> (AGUSTIN) virt_caps              : hvm hvm_directio
>
> (OLIVIER) virt_caps              : pv hvm hvm_directio pv_directio hap
> shadow iommu_hap_pt_share
>
>
> Could this be the problem?
>
>
> Agustín
>
>
> El 3/5/20 a las 18:50, Rob Townley escribió:
>> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
>> so increasing RAM allocated to Dom0 might speed up the VMs.
>>
>> 2GB for dom0 is extremely low in my opinion especially when most of
>> the 256GB of host RAM is going to waste.
>>
>> dom0_mem=2048M,max:4065M
>>
>> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert
>> <lambert.olivier@gmail.com <mailto:lambert.olivier@gmail.com>> wrote:
>>
>> Hard to tell. Here is my xl info to compare:
>>
>> host                   : xcp-ng-lab-3
>> release                : 4.19.0+1
>> version                : #1 SMP Thu Feb 13 17:34:28 CET 2020
>> machine                : x86_64
>> nr_cpus                : 4
>> max_cpu_id             : 3
>> nr_nodes               : 1
>> cores_per_socket       : 4
>> threads_per_core       : 1
>> cpu_mhz                : 3312.134
>> hw_caps                :
>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>> virt_caps              : pv hvm hvm_directio pv_directio hap
>> shadow iommu_hap_pt_share
>> total_memory           : 32634
>> free_memory            : 23619
>> sharing_freed_memory   : 0
>> sharing_used_memory    : 0
>> outstanding_claims     : 0
>> free_cpus              : 0
>> xen_major              : 4
>> xen_minor              : 13
>> xen_extra              : .0-8.4.xcpng8.1
>> xen_version            : 4.13.0-8.4.xcpng8.1
>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
>> hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>> xen_scheduler          : credit
>> xen_pagesize           : 4096
>> platform_params        : virt_start=0xffff800000000000
>> xen_changeset          : 85e1424de2dd, pq f9dbf852550e
>> xen_commandline        : watchdog ucode=scan dom0_max_vcpus=1-4
>> crashkernel=256M,below=4G console=vga vga=mode-0x0311
>> dom0_mem=8192M,max:8192M
>> cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
>> cc_compile_by          : mockbuild
>> cc_compile_domain      : [unknown]
>> cc_compile_date        : Tue Apr 14 18:28:14 CEST 2020
>> build_id               : 5ad6f12499d7f264544b64568b378260cd82a65f
>> xend_config_format     : 4
>>
>> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you.
>> So I ran the test on another server (building a VM just for you
>> :p ) and here is the result for a Xeon E5-2650L v2 @ 1.70GHz
>> (slow!) and VM disk stored on a NFS share.
>>
>> real 0m5,925s
>> user 0m3,769s
>> sys 0m2,321s
>>
>> Still, far better than 20 seconds you have!
>>
>> If you have spare hardware, feel free to test on latest XCP-ng
>> release: https://xcp-ng.org/docs/install.html
>>
>> Let me know if you need further help :)
>>
>>
>> Best,
>>
>> Olivier.
>>
>>
>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es
>> <mailto:Agustin.Lopez@uv.es>> a écrit :
>>
>> Hi Oliver.
>>
>> I am testing a bit more. In seconds, the results of the
>> command is:
>> Debian Buster PV -> 18'
>> Debian Buster HVM -> 8'
>> Debian Buster PVHVM -> 8'
>> Debian Buster PVH -> 8'
>>
>>
>> xl info
>> release                : 4.19.0-8-amd64
>> version                : #1 SMP Debian 4.19.98-1+deb10u1
>> (2020-04-27)
>> machine                : x86_64
>> nr_cpus                : 48
>> max_cpu_id             : 47
>> nr_nodes               : 2
>> cores_per_socket       : 12
>> threads_per_core       : 2
>> cpu_mhz                : 2197.458
>> hw_caps                :
>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>> virt_caps              : hvm hvm_directio
>> total_memory           : 261890
>> free_memory            : 255453
>> sharing_freed_memory   : 0
>> sharing_used_memory    : 0
>> outstanding_claims     : 0
>> free_cpus              : 0
>> xen_major              : 4
>> xen_minor              : 11
>> xen_extra              : .4-pre
>> xen_version            : 4.11.4-pre
>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
>> hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>> xen_scheduler          : credit
>> xen_pagesize           : 4096
>> platform_params        : virt_start=0xffff800000000000
>> xen_changeset          :
>> xen_commandline        : placeholder dom0_mem=2048M,max:4065M
>> cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
>> cc_compile_by          : pkg-xen-devel
>> cc_compile_domain      : lists.alioth.debian.org
>> <http://lists.alioth.debian.org>
>> cc_compile_date        : Wed Jan  8 20:16:51 UTC 2020
>> build_id               : b6822aa1d8f867753b92985e5cb0e806e520a08c
>> xend_config_format     : 4
>>
>> Oliver, I got > double values than you. Where is the problem?
>>
>> Regards,
>>
>> Agustín
>>
>>
>>
>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>> Hi Agustin,
>>>
>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh
>>> Debian 10 VM, and here is the result I have:
>>>
>>> ```
>>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f
>>> "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut -d" "
>>> -f5,9 >/dev/null
>>>
>>> real 0m2,741s
>>> user 0m2,248s
>>> sys 0m0,574s
>>> ```
>>>
>>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5
>>> (3.3Ghz) on a small Dell T30 machine, VM storage on local
>>> HDD. I did the test 3 times, and I have always results
>>> between 2,6 and 2,8 secs.
>>>
>>> Regards,
>>>
>>> Olivier.
>>>
>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez
>>> <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a écrit :
>>>
>>> Hello.
>>>
>>>
>>> We are testing low performance in IO with the next
>>> command in Debian Buster (kernel 4.19.0-8-amd64) with
>>> Xen (4.11.4-pre)
>>>
>>>         time for i in `dpkg -L ncurses-term | sort`; do
>>> if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "|
>>> cut -d" " -f5,9 >/dev/null
>>>
>>>
>>> In all our Dom0s - DomUs  we are getting around 20 seconds.
>>>
>>> In the same physical machines booting with Debian
>>> without Xen, we get 5-7 seconds
>>>
>>> In some KVM VMs in other server we are geting almost the
>>> same as physical.
>>>
>>> (all in local Disks. XFS filesystems. Images of DomUs in
>>> raw format)
>>>
>>>
>>> I have booted Xen with 4.8 y 4.4 releases with almost
>>> the same bad data.
>>>
>>>
>>> Where could be the problem?
>>>
>>> I think of is not normal this difference between DomUs
>>> and physical machine.
>>>
>>>
>>> Every pointer will be welcomed.
>>>
>>>
>>> Best regards,
>>>
>>> Agustín
>>>
>>>
>>>
>>>
>>>
Re: Bad performance with Xen [ In reply to ]
Hi!

Can you share your exact benchmark command so I can test it on my end?

Le lun. 4 mai 2020 à 10:25, GD <g.d.monnezza@tiscali.it> a écrit :

> Hi guys. Maybe we are suffering some related issue. If not, feel free to
> ignore this message.
> I wrote on this list but none replyed:
>
> "Fresh installed server with Debian Buster on top of nvme swRaid1 (mdadm)
> Testing hdd write seed with dd (with convert=fdatasync option) gives me
> the result of 330MB/s. Good.
> Installed xen-system and xen-tools (with --no-recommends option in apt)
> from official repository. Rebooted the system.
> Re-tested hdd write seed with dd (with convert=fdatasync option) gives me
> the result of 108MB/s. Not good at all.
>
> Maybe the following is not related to the issue, but on dmesg there is a
> line when I boot the system with Xen kernel:
> ...
> [ 14.214044] Performance Events: unsupported p6 CPU model 158 no PMU
> driver, software events only.
> ...
>
> Instead, when I boot the system without Xen kernel I have these lines in
> dmesg:
> ...
> [ 0.517217] Performance Events: PEBS fmt3+, Skylake events, 32-deep
> LBR, full-width counters, Intel PMU driver.
> [ 0.517356] ... version: 4
> [ 0.517444] ... bit width: 48
> [ 0.517444] ... generic registers: 4
> [ 0.517444] ... value mask: 0000ffffffffffff
> [ 0.517444] ... max period: 00007fffffffffff
> [ 0.517444] ... fixed-purpose events: 3
> [ 0.517444] ... event mask: 000000070000000f
>
> "
>
> Personally, I moved to KVM+libvirt nearly without rework.
> I/O performance are great.
> But I love XEN and I will be pleased to come back to it.
> g
>
>
> On 03/05/20 19:24, Agustin Lopez wrote:
>
>
> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>
> I have seen one difference between the 2 xl info:
>
> (AGUSTIN) virt_caps : hvm hvm_directio
>
> (OLIVIER) virt_caps : pv hvm hvm_directio pv_directio hap
> shadow iommu_hap_pt_share
>
>
> Could this be the problem?
>
>
> Agustín
>
>
> El 3/5/20 a las 18:50, Rob Townley escribió:
>
> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
> so increasing RAM allocated to Dom0 might speed up the VMs.
>
> 2GB for dom0 is extremely low in my opinion especially when most of the
> 256GB of host RAM is going to waste.
>
> dom0_mem=2048M,max:4065M
>
> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert <lambert.olivier@gmail.com>
> wrote:
>
>> Hard to tell. Here is my xl info to compare:
>>
>> host : xcp-ng-lab-3
>> release : 4.19.0+1
>> version : #1 SMP Thu Feb 13 17:34:28 CET 2020
>> machine : x86_64
>> nr_cpus : 4
>> max_cpu_id : 3
>> nr_nodes : 1
>> cores_per_socket : 4
>> threads_per_core : 1
>> cpu_mhz : 3312.134
>> hw_caps :
>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>> virt_caps : pv hvm hvm_directio pv_directio hap shadow
>> iommu_hap_pt_share
>> total_memory : 32634
>> free_memory : 23619
>> sharing_freed_memory : 0
>> sharing_used_memory : 0
>> outstanding_claims : 0
>> free_cpus : 0
>> xen_major : 4
>> xen_minor : 13
>> xen_extra : .0-8.4.xcpng8.1
>> xen_version : 4.13.0-8.4.xcpng8.1
>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>> hvm-3.0-x86_32p hvm-3.0-x86_64
>> xen_scheduler : credit
>> xen_pagesize : 4096
>> platform_params : virt_start=0xffff800000000000
>> xen_changeset : 85e1424de2dd, pq f9dbf852550e
>> xen_commandline : watchdog ucode=scan dom0_max_vcpus=1-4
>> crashkernel=256M,below=4G console=vga vga=mode-0x0311
>> dom0_mem=8192M,max:8192M
>> cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
>> cc_compile_by : mockbuild
>> cc_compile_domain : [unknown]
>> cc_compile_date : Tue Apr 14 18:28:14 CEST 2020
>> build_id : 5ad6f12499d7f264544b64568b378260cd82a65f
>> xend_config_format : 4
>>
>> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I ran
>> the test on another server (building a VM just for you :p ) and here is the
>> result for a Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored on a NFS
>> share.
>>
>> real 0m5,925s
>> user 0m3,769s
>> sys 0m2,321s
>>
>> Still, far better than 20 seconds you have!
>>
>> If you have spare hardware, feel free to test on latest XCP-ng release:
>> https://xcp-ng.org/docs/install.html
>>
>> Let me know if you need further help :)
>>
>>
>> Best,
>>
>> Olivier.
>>
>>
>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es> a écrit :
>>
>>> Hi Oliver.
>>>
>>> I am testing a bit more. In seconds, the results of the command is:
>>> Debian Buster PV -> 18'
>>> Debian Buster HVM -> 8'
>>> Debian Buster PVHVM -> 8'
>>> Debian Buster PVH -> 8'
>>>
>>>
>>> xl info
>>> release : 4.19.0-8-amd64
>>> version : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
>>> machine : x86_64
>>> nr_cpus : 48
>>> max_cpu_id : 47
>>> nr_nodes : 2
>>> cores_per_socket : 12
>>> threads_per_core : 2
>>> cpu_mhz : 2197.458
>>> hw_caps :
>>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>>> virt_caps : hvm hvm_directio
>>> total_memory : 261890
>>> free_memory : 255453
>>> sharing_freed_memory : 0
>>> sharing_used_memory : 0
>>> outstanding_claims : 0
>>> free_cpus : 0
>>> xen_major : 4
>>> xen_minor : 11
>>> xen_extra : .4-pre
>>> xen_version : 4.11.4-pre
>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_scheduler : credit
>>> xen_pagesize : 4096
>>> platform_params : virt_start=0xffff800000000000
>>> xen_changeset :
>>> xen_commandline : placeholder dom0_mem=2048M,max:4065M
>>> cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
>>> cc_compile_by : pkg-xen-devel
>>> cc_compile_domain : lists.alioth.debian.org
>>> cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
>>> build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
>>> xend_config_format : 4
>>>
>>> Oliver, I got > double values than you. Where is the problem?
>>>
>>> Regards,
>>>
>>> Agustín
>>>
>>>
>>>
>>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>>
>>> Hi Agustin,
>>>
>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM,
>>> and here is the result I have:
>>>
>>> ```
>>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then
>>> ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>
>>> real 0m2,741s
>>> user 0m2,248s
>>> sys 0m0,574s
>>> ```
>>>
>>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a
>>> small Dell T30 machine, VM storage on local HDD. I did the test 3 times,
>>> and I have always results between 2,6 and 2,8 secs.
>>>
>>> Regards,
>>>
>>> Olivier.
>>>
>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a
>>> écrit :
>>>
>>>> Hello.
>>>>
>>>>
>>>> We are testing low performance in IO with the next command in Debian
>>>> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>>>
>>>> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ];
>>>> then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>
>>>>
>>>> In all our Dom0s - DomUs we are getting around 20 seconds.
>>>>
>>>> In the same physical machines booting with Debian without Xen, we get
>>>> 5-7 seconds
>>>>
>>>> In some KVM VMs in other server we are geting almost the same as
>>>> physical.
>>>>
>>>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>>>
>>>>
>>>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad data.
>>>>
>>>>
>>>> Where could be the problem?
>>>>
>>>> I think of is not normal this difference between DomUs and physical
>>>> machine.
>>>>
>>>>
>>>> Every pointer will be welcomed.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Agustín
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>
Re: Bad performance with Xen [ In reply to ]
This is the command:
# dd bs=512 count=4194304 if=/dev/zero of=test conv=fdatasync
It creates a zero-filled file called "test" in the directory where the
command is executed.
Hope it helps

g

On 04/05/20 11:50, Olivier Lambert wrote:
> Hi!
>
> Can you share your exact benchmark command so I can test it on my end?
>
> Le lun. 4 mai 2020 à 10:25, GD <g.d.monnezza@tiscali.it
> <mailto:g.d.monnezza@tiscali.it>> a écrit :
>
> Hi guys. Maybe we are suffering some related issue. If not, feel
> free to ignore this message.
> I wrote on this list but none replyed:
>
> "Fresh installed server with Debian Buster on top of nvme swRaid1
> (mdadm)
> Testing hdd write seed with dd (with convert=fdatasync option)
> gives me the result of 330MB/s. Good.
> Installed xen-system and xen-tools (with --no-recommends option in
> apt) from official repository. Rebooted the system.
> Re-tested hdd write seed with dd (with convert=fdatasync option)
> gives me the result of 108MB/s. Not good at all.
>
> Maybe the following is not related to the issue, but on dmesg
> there is a line when I boot the system with Xen kernel:
> ...
> [   14.214044] Performance Events: unsupported p6 CPU model 158 no
> PMU driver, software events only.
> ...
>
> Instead, when I boot the system without Xen kernel I have these
> lines in dmesg:
> ...
> [    0.517217] Performance Events: PEBS fmt3+, Skylake events,
> 32-deep LBR, full-width counters, Intel PMU driver.
> [    0.517356] ... version:                4
> [    0.517444] ... bit width:              48
> [    0.517444] ... generic registers:      4
> [    0.517444] ... value mask: 0000ffffffffffff
> [    0.517444] ... max period: 00007fffffffffff
> [    0.517444] ... fixed-purpose events:   3
> [    0.517444] ... event mask: 000000070000000f
>
> "
>
> Personally, I moved to KVM+libvirt nearly without rework.
> I/O performance are great.
> But I love XEN and I will be pleased to come back to it.
> g
>
>
> On 03/05/20 19:24, Agustin Lopez wrote:
>>
>>
>> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>>
>> I have seen one difference between the 2 xl info:
>>
>> (AGUSTIN) virt_caps              : hvm hvm_directio
>>
>> (OLIVIER) virt_caps              : pv hvm hvm_directio
>> pv_directio hap shadow iommu_hap_pt_share
>>
>>
>> Could this be the problem?
>>
>>
>> Agustín
>>
>>
>> El 3/5/20 a las 18:50, Rob Townley escribió:
>>> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
>>> so increasing RAM allocated to Dom0 might speed up the VMs.
>>>
>>> 2GB for dom0 is extremely low in my opinion especially when most
>>> of the 256GB of host RAM is going to waste.
>>>
>>> dom0_mem=2048M,max:4065M
>>>
>>> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert
>>> <lambert.olivier@gmail.com <mailto:lambert.olivier@gmail.com>>
>>> wrote:
>>>
>>> Hard to tell. Here is my xl info to compare:
>>>
>>> host                   : xcp-ng-lab-3
>>> release                : 4.19.0+1
>>> version                : #1 SMP Thu Feb 13 17:34:28 CET 2020
>>> machine                : x86_64
>>> nr_cpus                : 4
>>> max_cpu_id             : 3
>>> nr_nodes               : 1
>>> cores_per_socket       : 4
>>> threads_per_core       : 1
>>> cpu_mhz                : 3312.134
>>> hw_caps                :
>>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>>> virt_caps              : pv hvm hvm_directio pv_directio hap
>>> shadow iommu_hap_pt_share
>>> total_memory           : 32634
>>> free_memory            : 23619
>>> sharing_freed_memory   : 0
>>> sharing_used_memory    : 0
>>> outstanding_claims     : 0
>>> free_cpus              : 0
>>> xen_major              : 4
>>> xen_minor              : 13
>>> xen_extra              : .0-8.4.xcpng8.1
>>> xen_version            : 4.13.0-8.4.xcpng8.1
>>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
>>> hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_scheduler          : credit
>>> xen_pagesize           : 4096
>>> platform_params        : virt_start=0xffff800000000000
>>> xen_changeset          : 85e1424de2dd, pq f9dbf852550e
>>> xen_commandline        : watchdog ucode=scan
>>> dom0_max_vcpus=1-4 crashkernel=256M,below=4G console=vga
>>> vga=mode-0x0311 dom0_mem=8192M,max:8192M
>>> cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat
>>> 4.8.5-28)
>>> cc_compile_by          : mockbuild
>>> cc_compile_domain      : [unknown]
>>> cc_compile_date        : Tue Apr 14 18:28:14 CEST 2020
>>> build_id               :
>>> 5ad6f12499d7f264544b64568b378260cd82a65f
>>> xend_config_format     : 4
>>>
>>> I'm on XCP-ng 8.1. Other diff is also I have more GHz than
>>> you. So I ran the test on another server (building a VM just
>>> for you :p ) and here is the result for a Xeon E5-2650L v2 @
>>> 1.70GHz (slow!) and VM disk stored on a NFS share.
>>>
>>> real 0m5,925s
>>> user 0m3,769s
>>> sys 0m2,321s
>>>
>>> Still, far better than 20 seconds you have!
>>>
>>> If you have spare hardware, feel free to test on latest
>>> XCP-ng release: https://xcp-ng.org/docs/install.html
>>>
>>> Let me know if you need further help :)
>>>
>>>
>>> Best,
>>>
>>> Olivier.
>>>
>>>
>>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez
>>> <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a écrit :
>>>
>>> Hi Oliver.
>>>
>>> I am testing a bit more. In seconds, the results of the
>>> command is:
>>> Debian Buster PV -> 18'
>>> Debian Buster HVM -> 8'
>>> Debian Buster PVHVM -> 8'
>>> Debian Buster PVH -> 8'
>>>
>>>
>>> xl info
>>> release                : 4.19.0-8-amd64
>>> version                : #1 SMP Debian 4.19.98-1+deb10u1
>>> (2020-04-27)
>>> machine                : x86_64
>>> nr_cpus                : 48
>>> max_cpu_id             : 47
>>> nr_nodes               : 2
>>> cores_per_socket       : 12
>>> threads_per_core       : 2
>>> cpu_mhz                : 2197.458
>>> hw_caps                :
>>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>>> virt_caps              : hvm hvm_directio
>>> total_memory           : 261890
>>> free_memory            : 255453
>>> sharing_freed_memory   : 0
>>> sharing_used_memory    : 0
>>> outstanding_claims     : 0
>>> free_cpus              : 0
>>> xen_major              : 4
>>> xen_minor              : 11
>>> xen_extra              : .4-pre
>>> xen_version            : 4.11.4-pre
>>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
>>> hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_scheduler          : credit
>>> xen_pagesize           : 4096
>>> platform_params        : virt_start=0xffff800000000000
>>> xen_changeset          :
>>> xen_commandline        : placeholder
>>> dom0_mem=2048M,max:4065M
>>> cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
>>> cc_compile_by          : pkg-xen-devel
>>> cc_compile_domain      : lists.alioth.debian.org
>>> <http://lists.alioth.debian.org>
>>> cc_compile_date        : Wed Jan  8 20:16:51 UTC 2020
>>> build_id               :
>>> b6822aa1d8f867753b92985e5cb0e806e520a08c
>>> xend_config_format     : 4
>>>
>>> Oliver, I got > double values than you. Where is the
>>> problem?
>>>
>>> Regards,
>>>
>>> Agustín
>>>
>>>
>>>
>>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>>> Hi Agustin,
>>>>
>>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh
>>>> Debian 10 VM, and here is the result I have:
>>>>
>>>> ```
>>>> # time for i in `dpkg -L ncurses-term | sort`; do if [
>>>> -f "$i" ]; then ls -ld  "$i"; fi; done | tr -s " "| cut
>>>> -d" " -f5,9 >/dev/null
>>>>
>>>> real 0m2,741s
>>>> user 0m2,248s
>>>> sys 0m0,574s
>>>> ```
>>>>
>>>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5
>>>> (3.3Ghz) on a small Dell T30 machine, VM storage on
>>>> local HDD. I did the test 3 times, and I have always
>>>> results between 2,6 and 2,8 secs.
>>>>
>>>> Regards,
>>>>
>>>> Olivier.
>>>>
>>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez
>>>> <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a
>>>> écrit :
>>>>
>>>> Hello.
>>>>
>>>>
>>>> We are testing low performance in IO with the next
>>>> command in Debian Buster (kernel 4.19.0-8-amd64)
>>>> with Xen (4.11.4-pre)
>>>>
>>>>         time for i in `dpkg -L ncurses-term |
>>>> sort`; do if [ -f "$i" ]; then ls -ld  "$i"; fi;
>>>> done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>
>>>>
>>>> In all our Dom0s - DomUs  we are getting around 20
>>>> seconds.
>>>>
>>>> In the same physical machines booting with Debian
>>>> without Xen, we get 5-7 seconds
>>>>
>>>> In some KVM VMs in other server we are geting
>>>> almost the same as physical.
>>>>
>>>> (all in local Disks. XFS filesystems. Images of
>>>> DomUs in raw format)
>>>>
>>>>
>>>> I have booted Xen with 4.8 y 4.4 releases with
>>>> almost the same bad data.
>>>>
>>>>
>>>> Where could be the problem?
>>>>
>>>> I think of is not normal this difference between
>>>> DomUs and physical machine.
>>>>
>>>>
>>>> Every pointer will be welcomed.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Agustín
>>>>
>>>>
>>>>
>>>>
>>>>
>
Re: Bad performance with Xen [ In reply to ]
I ran your test and I have 235 MB/s on a old Samsung SATA SSD (still on
XCP-ng 8.1 and a Debian VM on top).

Le lun. 4 mai 2020 à 12:33, GD <g.d.monnezza@tiscali.it> a écrit :

> This is the command:
> # dd bs=512 count=4194304 if=/dev/zero of=test conv=fdatasync
> It creates a zero-filled file called "test" in the directory where the
> command is executed.
> Hope it helps
>
> g
>
> On 04/05/20 11:50, Olivier Lambert wrote:
>
> Hi!
>
> Can you share your exact benchmark command so I can test it on my end?
>
> Le lun. 4 mai 2020 à 10:25, GD <g.d.monnezza@tiscali.it> a écrit :
>
>> Hi guys. Maybe we are suffering some related issue. If not, feel free to
>> ignore this message.
>> I wrote on this list but none replyed:
>>
>> "Fresh installed server with Debian Buster on top of nvme swRaid1 (mdadm)
>> Testing hdd write seed with dd (with convert=fdatasync option) gives me
>> the result of 330MB/s. Good.
>> Installed xen-system and xen-tools (with --no-recommends option in apt)
>> from official repository. Rebooted the system.
>> Re-tested hdd write seed with dd (with convert=fdatasync option) gives
>> me the result of 108MB/s. Not good at all.
>>
>> Maybe the following is not related to the issue, but on dmesg there is a
>> line when I boot the system with Xen kernel:
>> ...
>> [ 14.214044] Performance Events: unsupported p6 CPU model 158 no PMU
>> driver, software events only.
>> ...
>>
>> Instead, when I boot the system without Xen kernel I have these lines in
>> dmesg:
>> ...
>> [ 0.517217] Performance Events: PEBS fmt3+, Skylake events, 32-deep
>> LBR, full-width counters, Intel PMU driver.
>> [ 0.517356] ... version: 4
>> [ 0.517444] ... bit width: 48
>> [ 0.517444] ... generic registers: 4
>> [ 0.517444] ... value mask: 0000ffffffffffff
>> [ 0.517444] ... max period: 00007fffffffffff
>> [ 0.517444] ... fixed-purpose events: 3
>> [ 0.517444] ... event mask: 000000070000000f
>>
>> "
>>
>> Personally, I moved to KVM+libvirt nearly without rework.
>> I/O performance are great.
>> But I love XEN and I will be pleased to come back to it.
>> g
>>
>>
>> On 03/05/20 19:24, Agustin Lopez wrote:
>>
>>
>> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>>
>> I have seen one difference between the 2 xl info:
>>
>> (AGUSTIN) virt_caps : hvm hvm_directio
>>
>> (OLIVIER) virt_caps : pv hvm hvm_directio pv_directio hap
>> shadow iommu_hap_pt_share
>>
>>
>> Could this be the problem?
>>
>>
>> Agustín
>>
>>
>> El 3/5/20 a las 18:50, Rob Townley escribió:
>>
>> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
>> so increasing RAM allocated to Dom0 might speed up the VMs.
>>
>> 2GB for dom0 is extremely low in my opinion especially when most of the
>> 256GB of host RAM is going to waste.
>>
>> dom0_mem=2048M,max:4065M
>>
>> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert <
>> lambert.olivier@gmail.com> wrote:
>>
>>> Hard to tell. Here is my xl info to compare:
>>>
>>> host : xcp-ng-lab-3
>>> release : 4.19.0+1
>>> version : #1 SMP Thu Feb 13 17:34:28 CET 2020
>>> machine : x86_64
>>> nr_cpus : 4
>>> max_cpu_id : 3
>>> nr_nodes : 1
>>> cores_per_socket : 4
>>> threads_per_core : 1
>>> cpu_mhz : 3312.134
>>> hw_caps :
>>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>>> virt_caps : pv hvm hvm_directio pv_directio hap shadow
>>> iommu_hap_pt_share
>>> total_memory : 32634
>>> free_memory : 23619
>>> sharing_freed_memory : 0
>>> sharing_used_memory : 0
>>> outstanding_claims : 0
>>> free_cpus : 0
>>> xen_major : 4
>>> xen_minor : 13
>>> xen_extra : .0-8.4.xcpng8.1
>>> xen_version : 4.13.0-8.4.xcpng8.1
>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_scheduler : credit
>>> xen_pagesize : 4096
>>> platform_params : virt_start=0xffff800000000000
>>> xen_changeset : 85e1424de2dd, pq f9dbf852550e
>>> xen_commandline : watchdog ucode=scan dom0_max_vcpus=1-4
>>> crashkernel=256M,below=4G console=vga vga=mode-0x0311
>>> dom0_mem=8192M,max:8192M
>>> cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
>>> cc_compile_by : mockbuild
>>> cc_compile_domain : [unknown]
>>> cc_compile_date : Tue Apr 14 18:28:14 CEST 2020
>>> build_id : 5ad6f12499d7f264544b64568b378260cd82a65f
>>> xend_config_format : 4
>>>
>>> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I ran
>>> the test on another server (building a VM just for you :p ) and here is the
>>> result for a Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored on a NFS
>>> share.
>>>
>>> real 0m5,925s
>>> user 0m3,769s
>>> sys 0m2,321s
>>>
>>> Still, far better than 20 seconds you have!
>>>
>>> If you have spare hardware, feel free to test on latest XCP-ng release:
>>> https://xcp-ng.org/docs/install.html
>>>
>>> Let me know if you need further help :)
>>>
>>>
>>> Best,
>>>
>>> Olivier.
>>>
>>>
>>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es> a
>>> écrit :
>>>
>>>> Hi Oliver.
>>>>
>>>> I am testing a bit more. In seconds, the results of the command is:
>>>> Debian Buster PV -> 18'
>>>> Debian Buster HVM -> 8'
>>>> Debian Buster PVHVM -> 8'
>>>> Debian Buster PVH -> 8'
>>>>
>>>>
>>>> xl info
>>>> release : 4.19.0-8-amd64
>>>> version : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
>>>> machine : x86_64
>>>> nr_cpus : 48
>>>> max_cpu_id : 47
>>>> nr_nodes : 2
>>>> cores_per_socket : 12
>>>> threads_per_core : 2
>>>> cpu_mhz : 2197.458
>>>> hw_caps :
>>>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>>>> virt_caps : hvm hvm_directio
>>>> total_memory : 261890
>>>> free_memory : 255453
>>>> sharing_freed_memory : 0
>>>> sharing_used_memory : 0
>>>> outstanding_claims : 0
>>>> free_cpus : 0
>>>> xen_major : 4
>>>> xen_minor : 11
>>>> xen_extra : .4-pre
>>>> xen_version : 4.11.4-pre
>>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>>> xen_scheduler : credit
>>>> xen_pagesize : 4096
>>>> platform_params : virt_start=0xffff800000000000
>>>> xen_changeset :
>>>> xen_commandline : placeholder dom0_mem=2048M,max:4065M
>>>> cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
>>>> cc_compile_by : pkg-xen-devel
>>>> cc_compile_domain : lists.alioth.debian.org
>>>> cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
>>>> build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
>>>> xend_config_format : 4
>>>>
>>>> Oliver, I got > double values than you. Where is the problem?
>>>>
>>>> Regards,
>>>>
>>>> Agustín
>>>>
>>>>
>>>>
>>>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>>>
>>>> Hi Agustin,
>>>>
>>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM,
>>>> and here is the result I have:
>>>>
>>>> ```
>>>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then
>>>> ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>
>>>> real 0m2,741s
>>>> user 0m2,248s
>>>> sys 0m0,574s
>>>> ```
>>>>
>>>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a
>>>> small Dell T30 machine, VM storage on local HDD. I did the test 3 times,
>>>> and I have always results between 2,6 and 2,8 secs.
>>>>
>>>> Regards,
>>>>
>>>> Olivier.
>>>>
>>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a
>>>> écrit :
>>>>
>>>>> Hello.
>>>>>
>>>>>
>>>>> We are testing low performance in IO with the next command in Debian
>>>>> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>>>>
>>>>> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i"
>>>>> ]; then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>>
>>>>>
>>>>> In all our Dom0s - DomUs we are getting around 20 seconds.
>>>>>
>>>>> In the same physical machines booting with Debian without Xen, we get
>>>>> 5-7 seconds
>>>>>
>>>>> In some KVM VMs in other server we are geting almost the same as
>>>>> physical.
>>>>>
>>>>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>>>>
>>>>>
>>>>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad
>>>>> data.
>>>>>
>>>>>
>>>>> Where could be the problem?
>>>>>
>>>>> I think of is not normal this difference between DomUs and physical
>>>>> machine.
>>>>>
>>>>>
>>>>> Every pointer will be welcomed.
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Agustín
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>
>
Re: Bad performance with Xen [ In reply to ]
If you ran the test on the Dom0 xen-ified environment, the result is good.
You can perform a cross-check test and run the same command with the
server booted in normal environment (not with xen kernel enabled)
If you get the same result,  it means you experience no HDD I/O
performance loss and that my issue is related to something else.

Please note I'm not using XCP, but pure xen distro packages on Debian
Buster.

g
On 04/05/20 12:48, Olivier Lambert wrote:
> I ran your test and I have 235 MB/s on a old Samsung SATA SSD (still
> on XCP-ng 8.1 and a Debian VM on top).
>
> Le lun. 4 mai 2020 à 12:33, GD <g.d.monnezza@tiscali.it
> <mailto:g.d.monnezza@tiscali.it>> a écrit :
>
> This is the command:
> # dd bs=512 count=4194304 if=/dev/zero of=test conv=fdatasync
> It creates a zero-filled file called "test" in the directory where
> the command is executed.
> Hope it helps
>
> g
>
> On 04/05/20 11:50, Olivier Lambert wrote:
>> Hi!
>>
>> Can you share your exact benchmark command so I can test it on my
>> end?
>>
>> Le lun. 4 mai 2020 à 10:25, GD <g.d.monnezza@tiscali.it
>> <mailto:g.d.monnezza@tiscali.it>> a écrit :
>>
>> Hi guys. Maybe we are suffering some related issue. If not,
>> feel free to ignore this message.
>> I wrote on this list but none replyed:
>>
>> "Fresh installed server with Debian Buster on top of nvme
>> swRaid1 (mdadm)
>> Testing hdd write seed with dd (with convert=fdatasync
>> option) gives me the result of 330MB/s. Good.
>> Installed xen-system and xen-tools (with --no-recommends
>> option in apt) from official repository. Rebooted the system.
>> Re-tested hdd write seed with dd (with convert=fdatasync
>> option) gives me the result of 108MB/s. Not good at all.
>>
>> Maybe the following is not related to the issue, but on dmesg
>> there is a line when I boot the system with Xen kernel:
>> ...
>> [   14.214044] Performance Events: unsupported p6 CPU model
>> 158 no PMU driver, software events only.
>> ...
>>
>> Instead, when I boot the system without Xen kernel I have
>> these lines in dmesg:
>> ...
>> [    0.517217] Performance Events: PEBS fmt3+, Skylake
>> events, 32-deep LBR, full-width counters, Intel PMU driver.
>> [    0.517356] ... version:                4
>> [    0.517444] ... bit width:              48
>> [    0.517444] ... generic registers:      4
>> [    0.517444] ... value mask: 0000ffffffffffff
>> [    0.517444] ... max period: 00007fffffffffff
>> [    0.517444] ... fixed-purpose events:   3
>> [    0.517444] ... event mask: 000000070000000f
>>
>> "
>>
>> Personally, I moved to KVM+libvirt nearly without rework.
>> I/O performance are great.
>> But I love XEN and I will be pleased to come back to it.
>> g
>>
>>
>> On 03/05/20 19:24, Agustin Lopez wrote:
>>>
>>>
>>> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>>>
>>> I have seen one difference between the 2 xl info:
>>>
>>> (AGUSTIN) virt_caps              : hvm hvm_directio
>>>
>>> (OLIVIER) virt_caps              : pv hvm hvm_directio
>>> pv_directio hap shadow iommu_hap_pt_share
>>>
>>>
>>> Could this be the problem?
>>>
>>>
>>> Agustín
>>>
>>>
>>> El 3/5/20 a las 18:50, Rob Townley escribió:
>>>> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
>>>> so increasing RAM allocated to Dom0 might speed up the VMs.
>>>>
>>>> 2GB for dom0 is extremely low in my opinion especially when
>>>> most of the 256GB of host RAM is going to waste.
>>>>
>>>> dom0_mem=2048M,max:4065M
>>>>
>>>> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert
>>>> <lambert.olivier@gmail.com
>>>> <mailto:lambert.olivier@gmail.com>> wrote:
>>>>
>>>> Hard to tell. Here is my xl info to compare:
>>>>
>>>> host                   : xcp-ng-lab-3
>>>> release                : 4.19.0+1
>>>> version                : #1 SMP Thu Feb 13 17:34:28 CET
>>>> 2020
>>>> machine                : x86_64
>>>> nr_cpus                : 4
>>>> max_cpu_id             : 3
>>>> nr_nodes               : 1
>>>> cores_per_socket       : 4
>>>> threads_per_core       : 1
>>>> cpu_mhz                : 3312.134
>>>> hw_caps                :
>>>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>>>> virt_caps              : pv hvm hvm_directio
>>>> pv_directio hap shadow iommu_hap_pt_share
>>>> total_memory           : 32634
>>>> free_memory            : 23619
>>>> sharing_freed_memory   : 0
>>>> sharing_used_memory    : 0
>>>> outstanding_claims     : 0
>>>> free_cpus              : 0
>>>> xen_major              : 4
>>>> xen_minor              : 13
>>>> xen_extra              : .0-8.4.xcpng8.1
>>>> xen_version            : 4.13.0-8.4.xcpng8.1
>>>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
>>>> hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>>>> xen_scheduler          : credit
>>>> xen_pagesize           : 4096
>>>> platform_params        : virt_start=0xffff800000000000
>>>> xen_changeset          : 85e1424de2dd, pq f9dbf852550e
>>>> xen_commandline        : watchdog ucode=scan
>>>> dom0_max_vcpus=1-4 crashkernel=256M,below=4G
>>>> console=vga vga=mode-0x0311 dom0_mem=8192M,max:8192M
>>>> cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red
>>>> Hat 4.8.5-28)
>>>> cc_compile_by          : mockbuild
>>>> cc_compile_domain      : [unknown]
>>>> cc_compile_date        : Tue Apr 14 18:28:14 CEST 2020
>>>> build_id               :
>>>> 5ad6f12499d7f264544b64568b378260cd82a65f
>>>> xend_config_format     : 4
>>>>
>>>> I'm on XCP-ng 8.1. Other diff is also I have more GHz
>>>> than you. So I ran the test on another server (building
>>>> a VM just for you :p ) and here is the result for a
>>>> Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored
>>>> on a NFS share.
>>>>
>>>> real 0m5,925s
>>>> user 0m3,769s
>>>> sys 0m2,321s
>>>>
>>>> Still, far better than 20 seconds you have!
>>>>
>>>> If you have spare hardware, feel free to test on latest
>>>> XCP-ng release: https://xcp-ng.org/docs/install.html
>>>>
>>>> Let me know if you need further help :)
>>>>
>>>>
>>>> Best,
>>>>
>>>> Olivier.
>>>>
>>>>
>>>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez
>>>> <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>> a
>>>> écrit :
>>>>
>>>> Hi Oliver.
>>>>
>>>> I am testing a bit more. In seconds, the results of
>>>> the command is:
>>>> Debian Buster PV -> 18'
>>>> Debian Buster HVM -> 8'
>>>> Debian Buster PVHVM -> 8'
>>>> Debian Buster PVH -> 8'
>>>>
>>>>
>>>> xl info
>>>> release                : 4.19.0-8-amd64
>>>> version                : #1 SMP Debian
>>>> 4.19.98-1+deb10u1 (2020-04-27)
>>>> machine                : x86_64
>>>> nr_cpus                : 48
>>>> max_cpu_id             : 47
>>>> nr_nodes               : 2
>>>> cores_per_socket       : 12
>>>> threads_per_core       : 2
>>>> cpu_mhz                : 2197.458
>>>> hw_caps                :
>>>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>>>> virt_caps              : hvm hvm_directio
>>>> total_memory           : 261890
>>>> free_memory            : 255453
>>>> sharing_freed_memory   : 0
>>>> sharing_used_memory    : 0
>>>> outstanding_claims     : 0
>>>> free_cpus              : 0
>>>> xen_major              : 4
>>>> xen_minor              : 11
>>>> xen_extra              : .4-pre
>>>> xen_version            : 4.11.4-pre
>>>> xen_caps               : xen-3.0-x86_64
>>>> xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
>>>> hvm-3.0-x86_64
>>>> xen_scheduler          : credit
>>>> xen_pagesize           : 4096
>>>> platform_params        : virt_start=0xffff800000000000
>>>> xen_changeset          :
>>>> xen_commandline        : placeholder
>>>> dom0_mem=2048M,max:4065M
>>>> cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
>>>> cc_compile_by          : pkg-xen-devel
>>>> cc_compile_domain      : lists.alioth.debian.org
>>>> <http://lists.alioth.debian.org>
>>>> cc_compile_date        : Wed Jan  8 20:16:51 UTC 2020
>>>> build_id               :
>>>> b6822aa1d8f867753b92985e5cb0e806e520a08c
>>>> xend_config_format     : 4
>>>>
>>>> Oliver, I got > double values than you. Where is
>>>> the problem?
>>>>
>>>> Regards,
>>>>
>>>> Agustín
>>>>
>>>>
>>>>
>>>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>>>> Hi Agustin,
>>>>>
>>>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a
>>>>> fresh Debian 10 VM, and here is the result I have:
>>>>>
>>>>> ```
>>>>> # time for i in `dpkg -L ncurses-term | sort`; do
>>>>> if [ -f "$i" ]; then ls -ld  "$i"; fi; done | tr
>>>>> -s " "| cut -d" " -f5,9 >/dev/null
>>>>>
>>>>> real 0m2,741s
>>>>> user 0m2,248s
>>>>> sys 0m0,574s
>>>>> ```
>>>>>
>>>>> My hardware isn't ultra modern: Xeon(R) CPU
>>>>> E3-1225 v5 (3.3Ghz) on a small Dell T30 machine,
>>>>> VM storage on local HDD. I did the test 3 times,
>>>>> and I have always results between 2,6 and 2,8 secs.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Olivier.
>>>>>
>>>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez
>>>>> <Agustin.Lopez@uv.es <mailto:Agustin.Lopez@uv.es>>
>>>>> a écrit :
>>>>>
>>>>> Hello.
>>>>>
>>>>>
>>>>> We are testing low performance in IO with the
>>>>> next command in Debian Buster (kernel
>>>>> 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>>>>
>>>>>         time for i in `dpkg -L ncurses-term |
>>>>> sort`; do if [ -f "$i" ]; then ls -ld  "$i";
>>>>> fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>>
>>>>>
>>>>> In all our Dom0s - DomUs  we are getting
>>>>> around 20 seconds.
>>>>>
>>>>> In the same physical machines booting with
>>>>> Debian without Xen, we get 5-7 seconds
>>>>>
>>>>> In some KVM VMs in other server we are geting
>>>>> almost the same as physical.
>>>>>
>>>>> (all in local Disks. XFS filesystems. Images
>>>>> of DomUs in raw format)
>>>>>
>>>>>
>>>>> I have booted Xen with 4.8 y 4.4 releases with
>>>>> almost the same bad data.
>>>>>
>>>>>
>>>>> Where could be the problem?
>>>>>
>>>>> I think of is not normal this difference
>>>>> between DomUs and physical machine.
>>>>>
>>>>>
>>>>> Every pointer will be welcomed.
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Agustín
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>
>
Re: Bad performance with Xen [ In reply to ]
Sorry but I don't have spare hardware to do so… If you have, please do test
XCP-ng in case, it's trivial to setup :)

Le lun. 4 mai 2020 à 14:40, GD <g.d.monnezza@tiscali.it> a écrit :

> If you ran the test on the Dom0 xen-ified environment, the result is good.
> You can perform a cross-check test and run the same command with the
> server booted in normal environment (not with xen kernel enabled)
> If you get the same result, it means you experience no HDD I/O performance
> loss and that my issue is related to something else.
>
> Please note I'm not using XCP, but pure xen distro packages on Debian
> Buster.
>
> g
> On 04/05/20 12:48, Olivier Lambert wrote:
>
> I ran your test and I have 235 MB/s on a old Samsung SATA SSD (still on
> XCP-ng 8.1 and a Debian VM on top).
>
> Le lun. 4 mai 2020 à 12:33, GD <g.d.monnezza@tiscali.it> a écrit :
>
>> This is the command:
>> # dd bs=512 count=4194304 if=/dev/zero of=test conv=fdatasync
>> It creates a zero-filled file called "test" in the directory where the
>> command is executed.
>> Hope it helps
>>
>> g
>>
>> On 04/05/20 11:50, Olivier Lambert wrote:
>>
>> Hi!
>>
>> Can you share your exact benchmark command so I can test it on my end?
>>
>> Le lun. 4 mai 2020 à 10:25, GD <g.d.monnezza@tiscali.it> a écrit :
>>
>>> Hi guys. Maybe we are suffering some related issue. If not, feel free to
>>> ignore this message.
>>> I wrote on this list but none replyed:
>>>
>>> "Fresh installed server with Debian Buster on top of nvme swRaid1
>>> (mdadm)
>>> Testing hdd write seed with dd (with convert=fdatasync option) gives me
>>> the result of 330MB/s. Good.
>>> Installed xen-system and xen-tools (with --no-recommends option in apt)
>>> from official repository. Rebooted the system.
>>> Re-tested hdd write seed with dd (with convert=fdatasync option) gives
>>> me the result of 108MB/s. Not good at all.
>>>
>>> Maybe the following is not related to the issue, but on dmesg there is a
>>> line when I boot the system with Xen kernel:
>>> ...
>>> [ 14.214044] Performance Events: unsupported p6 CPU model 158 no PMU
>>> driver, software events only.
>>> ...
>>>
>>> Instead, when I boot the system without Xen kernel I have these lines in
>>> dmesg:
>>> ...
>>> [ 0.517217] Performance Events: PEBS fmt3+, Skylake events, 32-deep
>>> LBR, full-width counters, Intel PMU driver.
>>> [ 0.517356] ... version: 4
>>> [ 0.517444] ... bit width: 48
>>> [ 0.517444] ... generic registers: 4
>>> [ 0.517444] ... value mask: 0000ffffffffffff
>>> [ 0.517444] ... max period: 00007fffffffffff
>>> [ 0.517444] ... fixed-purpose events: 3
>>> [ 0.517444] ... event mask: 000000070000000f
>>>
>>> "
>>>
>>> Personally, I moved to KVM+libvirt nearly without rework.
>>> I/O performance are great.
>>> But I love XEN and I will be pleased to come back to it.
>>> g
>>>
>>>
>>> On 03/05/20 19:24, Agustin Lopez wrote:
>>>
>>>
>>> Sorry. I booted with 8 GB for the Dom0 and all is the same.
>>>
>>> I have seen one difference between the 2 xl info:
>>>
>>> (AGUSTIN) virt_caps : hvm hvm_directio
>>>
>>> (OLIVIER) virt_caps : pv hvm hvm_directio pv_directio hap
>>> shadow iommu_hap_pt_share
>>>
>>>
>>> Could this be the problem?
>>>
>>>
>>> Agustín
>>>
>>>
>>> El 3/5/20 a las 18:50, Rob Townley escribió:
>>>
>>> Agustin, noticed ‘ dom0_mem=2048M,max:4065M’,
>>> so increasing RAM allocated to Dom0 might speed up the VMs.
>>>
>>> 2GB for dom0 is extremely low in my opinion especially when most of the
>>> 256GB of host RAM is going to waste.
>>>
>>> dom0_mem=2048M,max:4065M
>>>
>>> On Sun, May 3, 2020 at 10:41 AM Olivier Lambert <
>>> lambert.olivier@gmail.com> wrote:
>>>
>>>> Hard to tell. Here is my xl info to compare:
>>>>
>>>> host : xcp-ng-lab-3
>>>> release : 4.19.0+1
>>>> version : #1 SMP Thu Feb 13 17:34:28 CET 2020
>>>> machine : x86_64
>>>> nr_cpus : 4
>>>> max_cpu_id : 3
>>>> nr_nodes : 1
>>>> cores_per_socket : 4
>>>> threads_per_core : 1
>>>> cpu_mhz : 3312.134
>>>> hw_caps :
>>>> bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
>>>> virt_caps : pv hvm hvm_directio pv_directio hap shadow
>>>> iommu_hap_pt_share
>>>> total_memory : 32634
>>>> free_memory : 23619
>>>> sharing_freed_memory : 0
>>>> sharing_used_memory : 0
>>>> outstanding_claims : 0
>>>> free_cpus : 0
>>>> xen_major : 4
>>>> xen_minor : 13
>>>> xen_extra : .0-8.4.xcpng8.1
>>>> xen_version : 4.13.0-8.4.xcpng8.1
>>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>>> xen_scheduler : credit
>>>> xen_pagesize : 4096
>>>> platform_params : virt_start=0xffff800000000000
>>>> xen_changeset : 85e1424de2dd, pq f9dbf852550e
>>>> xen_commandline : watchdog ucode=scan dom0_max_vcpus=1-4
>>>> crashkernel=256M,below=4G console=vga vga=mode-0x0311
>>>> dom0_mem=8192M,max:8192M
>>>> cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
>>>> cc_compile_by : mockbuild
>>>> cc_compile_domain : [unknown]
>>>> cc_compile_date : Tue Apr 14 18:28:14 CEST 2020
>>>> build_id : 5ad6f12499d7f264544b64568b378260cd82a65f
>>>> xend_config_format : 4
>>>>
>>>> I'm on XCP-ng 8.1. Other diff is also I have more GHz than you. So I
>>>> ran the test on another server (building a VM just for you :p ) and here is
>>>> the result for a Xeon E5-2650L v2 @ 1.70GHz (slow!) and VM disk stored on a
>>>> NFS share.
>>>>
>>>> real 0m5,925s
>>>> user 0m3,769s
>>>> sys 0m2,321s
>>>>
>>>> Still, far better than 20 seconds you have!
>>>>
>>>> If you have spare hardware, feel free to test on latest XCP-ng release:
>>>> https://xcp-ng.org/docs/install.html
>>>>
>>>> Let me know if you need further help :)
>>>>
>>>>
>>>> Best,
>>>>
>>>> Olivier.
>>>>
>>>>
>>>> Le dim. 3 mai 2020 à 14:27, Agustin Lopez <Agustin.Lopez@uv.es> a
>>>> écrit :
>>>>
>>>>> Hi Oliver.
>>>>>
>>>>> I am testing a bit more. In seconds, the results of the command is:
>>>>> Debian Buster PV -> 18'
>>>>> Debian Buster HVM -> 8'
>>>>> Debian Buster PVHVM -> 8'
>>>>> Debian Buster PVH -> 8'
>>>>>
>>>>>
>>>>> xl info
>>>>> release : 4.19.0-8-amd64
>>>>> version : #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
>>>>> machine : x86_64
>>>>> nr_cpus : 48
>>>>> max_cpu_id : 47
>>>>> nr_nodes : 2
>>>>> cores_per_socket : 12
>>>>> threads_per_core : 2
>>>>> cpu_mhz : 2197.458
>>>>> hw_caps :
>>>>> bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
>>>>> virt_caps : hvm hvm_directio
>>>>> total_memory : 261890
>>>>> free_memory : 255453
>>>>> sharing_freed_memory : 0
>>>>> sharing_used_memory : 0
>>>>> outstanding_claims : 0
>>>>> free_cpus : 0
>>>>> xen_major : 4
>>>>> xen_minor : 11
>>>>> xen_extra : .4-pre
>>>>> xen_version : 4.11.4-pre
>>>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>>>> xen_scheduler : credit
>>>>> xen_pagesize : 4096
>>>>> platform_params : virt_start=0xffff800000000000
>>>>> xen_changeset :
>>>>> xen_commandline : placeholder dom0_mem=2048M,max:4065M
>>>>> cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
>>>>> cc_compile_by : pkg-xen-devel
>>>>> cc_compile_domain : lists.alioth.debian.org
>>>>> cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
>>>>> build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
>>>>> xend_config_format : 4
>>>>>
>>>>> Oliver, I got > double values than you. Where is the problem?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Agustín
>>>>>
>>>>>
>>>>>
>>>>> El 2/5/20 a las 19:56, Olivier Lambert escribió:
>>>>>
>>>>> Hi Agustin,
>>>>>
>>>>> I just did a test on XCP-ng 8.1 (Xen 4.13) with a fresh Debian 10 VM,
>>>>> and here is the result I have:
>>>>>
>>>>> ```
>>>>> # time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i" ]; then
>>>>> ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>>
>>>>> real 0m2,741s
>>>>> user 0m2,248s
>>>>> sys 0m0,574s
>>>>> ```
>>>>>
>>>>> My hardware isn't ultra modern: Xeon(R) CPU E3-1225 v5 (3.3Ghz) on a
>>>>> small Dell T30 machine, VM storage on local HDD. I did the test 3 times,
>>>>> and I have always results between 2,6 and 2,8 secs.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Olivier.
>>>>>
>>>>> Le sam. 2 mai 2020 à 18:33, Agustin Lopez <Agustin.Lopez@uv.es> a
>>>>> écrit :
>>>>>
>>>>>> Hello.
>>>>>>
>>>>>>
>>>>>> We are testing low performance in IO with the next command in Debian
>>>>>> Buster (kernel 4.19.0-8-amd64) with Xen (4.11.4-pre)
>>>>>>
>>>>>> time for i in `dpkg -L ncurses-term | sort`; do if [ -f "$i"
>>>>>> ]; then ls -ld "$i"; fi; done | tr -s " "| cut -d" " -f5,9 >/dev/null
>>>>>>
>>>>>>
>>>>>> In all our Dom0s - DomUs we are getting around 20 seconds.
>>>>>>
>>>>>> In the same physical machines booting with Debian without Xen, we get
>>>>>> 5-7 seconds
>>>>>>
>>>>>> In some KVM VMs in other server we are geting almost the same as
>>>>>> physical.
>>>>>>
>>>>>> (all in local Disks. XFS filesystems. Images of DomUs in raw format)
>>>>>>
>>>>>>
>>>>>> I have booted Xen with 4.8 y 4.4 releases with almost the same bad
>>>>>> data.
>>>>>>
>>>>>>
>>>>>> Where could be the problem?
>>>>>>
>>>>>> I think of is not normal this difference between DomUs and physical
>>>>>> machine.
>>>>>>
>>>>>>
>>>>>> Every pointer will be welcomed.
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Agustín
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>
>