TLDR version: Setting "-E 10000" for xenstored and rebooting solved the issue.
On 3 June 2020 15:26:19 CEST, "Jürgen Groß" <jgross@suse.com> wrote:
>On 03.06.20 15:01, J. Roeleveld wrote:
>> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
>>> On 03.06.20 14:15, J. Roeleveld wrote:
>>>> Hi all,
>>>>
>>>> I upgraded to Xen 4.12.2 recently and since the upgrade, I am
>unable to
>>>> start more than 14 domUs. >
>>>> The error, when trying to start the 15th, is:
>>>> Parsing config from /etc/xen/boot/stage7/dbtest
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain
>20:unable to
>>>> add disk devices
>>>> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
>>>> 20:Non-existant domain
>>>> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain
>20:Unable
>>>> to destroy guest
>>>> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain
>20:Destruction
>>>> of domain failed
>>>>
>>>>
>>>>
>>>> The domUs showing the issue get the disks from a driver-domain.
>>>
>>> Ah, this might be related to hitting some Xenstore quota.
>>>
>>> As you are seeing the problem in libxl__xs_transaction_commit I
>guess
>>> you should increase the number of Xenstore entries per domain.
>>>
>>> The default is 1000, you can set the value via the -E start
>parameter of
>>> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
>>> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
>>> "quota-maxentity").
>>
>> Has this default been changed/lowered?
>> Or has there been an increase in amount of values in xenstore?
>
>Depends. Which Xen version did you use before, and which Xenstore
>variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
>Did you change anything in the driver domain?
I need to check the logs to see which version it was. I wasn't that far behind in version numbers.
I am running Gentoo as my OS and afaik, I only have 1 option: "xenstored" not heard of the other options yet, will do some research. (If you have some bookmarks, I would appreciate it. Otherwise it'll be whatever I can find)
I already noticed the issue before updating the driver domain. Initially I suspected a version mismatch and waited for a chance to update the driver domain. As that also didn't solve it and I still couldn't find anything usefull online, I asked for help here.
>> As this will require a reboot of the server, I will not be able to
>test this
>> before the weekend. Is there any way to confirm this is actually the
>case
>> without a restart?
>
>There are strong hints in the data you presented:
>
>libxl__xs_transaction_commit() is only calling xs_transaction_end()
>and the error you are seeing is ENOSPC. In transaction end handling
>the only reason for ENOSPC is hitting the quota for the number of
>Xenstore nodes per domain.
>
>You can easily check this by calling
>
>xenstore-ls -p
>
>in dom0 and count the number of entries which are owned by the driver
>domain, an entry looks like:
>
> console = "" . . . . . . . . . . . . . . . . . . . . . (n0,n1)
>
>where "(n0,n1)" are the permissions of the entry. The first permission
>entry "n0" is the owner (drop the letter, and you get the domid of the
>owner, in my example "0").
I will investigate the output later this week. Did see a strong hint at the "1000" value. The "ldb" file for xenstored was 1000kb in size.
I set the option to "-E 10000" to be certain I won't have this issue again for a long time and managed to organize a slot to reboot the system.
And, thank you so very much, the issue has now been solved.
Many thanks,
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 3 June 2020 15:26:19 CEST, "Jürgen Groß" <jgross@suse.com> wrote:
>On 03.06.20 15:01, J. Roeleveld wrote:
>> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
>>> On 03.06.20 14:15, J. Roeleveld wrote:
>>>> Hi all,
>>>>
>>>> I upgraded to Xen 4.12.2 recently and since the upgrade, I am
>unable to
>>>> start more than 14 domUs. >
>>>> The error, when trying to start the 15th, is:
>>>> Parsing config from /etc/xen/boot/stage7/dbtest
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain
>20:unable to
>>>> add disk devices
>>>> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
>>>> 20:Non-existant domain
>>>> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain
>20:Unable
>>>> to destroy guest
>>>> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain
>20:Destruction
>>>> of domain failed
>>>>
>>>>
>>>>
>>>> The domUs showing the issue get the disks from a driver-domain.
>>>
>>> Ah, this might be related to hitting some Xenstore quota.
>>>
>>> As you are seeing the problem in libxl__xs_transaction_commit I
>guess
>>> you should increase the number of Xenstore entries per domain.
>>>
>>> The default is 1000, you can set the value via the -E start
>parameter of
>>> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
>>> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
>>> "quota-maxentity").
>>
>> Has this default been changed/lowered?
>> Or has there been an increase in amount of values in xenstore?
>
>Depends. Which Xen version did you use before, and which Xenstore
>variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
>Did you change anything in the driver domain?
I need to check the logs to see which version it was. I wasn't that far behind in version numbers.
I am running Gentoo as my OS and afaik, I only have 1 option: "xenstored" not heard of the other options yet, will do some research. (If you have some bookmarks, I would appreciate it. Otherwise it'll be whatever I can find)
I already noticed the issue before updating the driver domain. Initially I suspected a version mismatch and waited for a chance to update the driver domain. As that also didn't solve it and I still couldn't find anything usefull online, I asked for help here.
>> As this will require a reboot of the server, I will not be able to
>test this
>> before the weekend. Is there any way to confirm this is actually the
>case
>> without a restart?
>
>There are strong hints in the data you presented:
>
>libxl__xs_transaction_commit() is only calling xs_transaction_end()
>and the error you are seeing is ENOSPC. In transaction end handling
>the only reason for ENOSPC is hitting the quota for the number of
>Xenstore nodes per domain.
>
>You can easily check this by calling
>
>xenstore-ls -p
>
>in dom0 and count the number of entries which are owned by the driver
>domain, an entry looks like:
>
> console = "" . . . . . . . . . . . . . . . . . . . . . (n0,n1)
>
>where "(n0,n1)" are the permissions of the entry. The first permission
>entry "n0" is the owner (drop the letter, and you get the domid of the
>owner, in my example "0").
I will investigate the output later this week. Did see a strong hint at the "1000" value. The "ldb" file for xenstored was 1000kb in size.
I set the option to "-E 10000" to be certain I won't have this issue again for a long time and managed to organize a slot to reboot the system.
And, thank you so very much, the issue has now been solved.
Many thanks,
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.