Mailing List Archive: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction

Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction

Jun 3, 2020, 5:15 AM

Post #1 of 6 (196 views)

Hi all,

I upgraded to Xen 4.12.2 recently and since the upgrade, I am unable to start
more than 14 domUs.

The error, when trying to start the 15th, is:
Parsing config from /etc/xen/boot/stage7/dbtest
libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
commit xenstore transaction: No space left on device
libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
commit xenstore transaction: No space left on device
libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain 20:unable to add
disk devices
libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain 20:Non-existant
domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 20:Unable to
destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 20:Destruction of
domain failed

The domUs showing the issue get the disks from a driver-domain.
All this was working correctly using older Xen versions and no config changes
have been done.

Unfortunately, I am unable to find anything further in the logs and my google-
fu is not helping either as all I find is for much older versions.

If anyone has any further ideas on how to further analyse this?

If I stop 1 domU, I am able to start another one, so I am confident it has to
do with amount of domUs running and not a specific domU.

Many thanks,

Joost

Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction [ In reply to ]

jgross at suse

Jun 3, 2020, 5:56 AM

Post #2 of 6 (196 views)

Permalink

On 03.06.20 14:15, J. Roeleveld wrote:
> Hi all,
>
> I upgraded to Xen 4.12.2 recently and since the upgrade, I am unable to start
> more than 14 domUs. >
> The error, when trying to start the 15th, is:
> Parsing config from /etc/xen/boot/stage7/dbtest
> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> commit xenstore transaction: No space left on device
> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> commit xenstore transaction: No space left on device
> libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain 20:unable to add
> disk devices
> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain 20:Non-existant
> domain
> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 20:Unable to
> destroy guest
> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 20:Destruction of
> domain failed
>
>
>
> The domUs showing the issue get the disks from a driver-domain.

Ah, this might be related to hitting some Xenstore quota.

As you are seeing the problem in libxl__xs_transaction_commit I guess
you should increase the number of Xenstore entries per domain.

The default is 1000, you can set the value via the -E start parameter of
xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
oxenstored by editing your /etc/xen/oxenstored.conf file (entry
"quota-maxentity").

Juergen

Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction [ In reply to ]

joost at antarean

Jun 3, 2020, 6:01 AM

Post #3 of 6 (196 views)

Permalink

On Wednesday, June 3, 2020 2:56:06 PM CEST J?rgen Gro? wrote:
> On 03.06.20 14:15, J. Roeleveld wrote:
> > Hi all,
> >
> > I upgraded to Xen 4.12.2 recently and since the upgrade, I am unable to
> > start more than 14 domUs. >
> > The error, when trying to start the 15th, is:
> > Parsing config from /etc/xen/boot/stage7/dbtest
> > libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> > commit xenstore transaction: No space left on device
> > libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> > commit xenstore transaction: No space left on device
> > libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain 20:unable to
> > add disk devices
> > libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
> > 20:Non-existant domain
> > libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 20:Unable
> > to destroy guest
> > libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 20:Destruction
> > of domain failed
> >
> >
> >
> > The domUs showing the issue get the disks from a driver-domain.
>
> Ah, this might be related to hitting some Xenstore quota.
>
> As you are seeing the problem in libxl__xs_transaction_commit I guess
> you should increase the number of Xenstore entries per domain.
>
> The default is 1000, you can set the value via the -E start parameter of
> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
> "quota-maxentity").

Has this default been changed/lowered?
Or has there been an increase in amount of values in xenstore?

As this will require a reboot of the server, I will not be able to test this
before the weekend. Is there any way to confirm this is actually the case
without a restart?

Many thanks,

Joost

Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction [ In reply to ]

jgross at suse

Jun 3, 2020, 6:26 AM

Post #4 of 6 (196 views)

Permalink

On 03.06.20 15:01, J. Roeleveld wrote:
> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
>> On 03.06.20 14:15, J. Roeleveld wrote:
>>> Hi all,
>>>
>>> I upgraded to Xen 4.12.2 recently and since the upgrade, I am unable to
>>> start more than 14 domUs. >
>>> The error, when trying to start the 15th, is:
>>> Parsing config from /etc/xen/boot/stage7/dbtest
>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
>>> commit xenstore transaction: No space left on device
>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
>>> commit xenstore transaction: No space left on device
>>> libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain 20:unable to
>>> add disk devices
>>> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
>>> 20:Non-existant domain
>>> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 20:Unable
>>> to destroy guest
>>> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 20:Destruction
>>> of domain failed
>>>
>>>
>>>
>>> The domUs showing the issue get the disks from a driver-domain.
>>
>> Ah, this might be related to hitting some Xenstore quota.
>>
>> As you are seeing the problem in libxl__xs_transaction_commit I guess
>> you should increase the number of Xenstore entries per domain.
>>
>> The default is 1000, you can set the value via the -E start parameter of
>> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
>> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
>> "quota-maxentity").
>
> Has this default been changed/lowered?
> Or has there been an increase in amount of values in xenstore?

Depends. Which Xen version did you use before, and which Xenstore
variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
Did you change anything in the driver domain?

> As this will require a reboot of the server, I will not be able to test this
> before the weekend. Is there any way to confirm this is actually the case
> without a restart?

There are strong hints in the data you presented:

libxl__xs_transaction_commit() is only calling xs_transaction_end()
and the error you are seeing is ENOSPC. In transaction end handling
the only reason for ENOSPC is hitting the quota for the number of
Xenstore nodes per domain.

You can easily check this by calling

xenstore-ls -p

in dom0 and count the number of entries which are owned by the driver
domain, an entry looks like:

console = "" . . . . . . . . . . . . . . . . . . . . . (n0,n1)

where "(n0,n1)" are the permissions of the entry. The first permission
entry "n0" is the owner (drop the letter, and you get the domid of the
owner, in my example "0").

Juergen

Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction [ In reply to ]

joost at antarean

Jun 4, 2020, 5:00 AM

Post #5 of 6 (192 views)

Permalink

On Wednesday, June 3, 2020 3:26:19 PM CEST J?rgen Gro? wrote:
> On 03.06.20 15:01, J. Roeleveld wrote:
> > On Wednesday, June 3, 2020 2:56:06 PM CEST J?rgen Gro? wrote:

> >> The default is 1000, you can set the value via the -E start parameter of
> >> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
> >> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
> >> "quota-maxentity").
> >
> > Has this default been changed/lowered?
> > Or has there been an increase in amount of values in xenstore?
>
> Depends. Which Xen version did you use before, and which Xenstore
> variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
> Did you change anything in the driver domain?

My previous version was 4.11.2.
Using xenstored.
No changes done to the driver domain.
- Issue occured initially when driver domain was still on 4.11.2
- Issue still occured when driver domain was upgraded to 4.12.2

In my opinion, putting a quota on the xenstored database at 1000kb is far too
low. I am not doing anything special and have had no issues running over 20
VMs simultaneously in the past.

Suddenly encountering this is unexpected and, if sane defaults are used,
unnecessary.

Do you happen to know if this limit was changed between 4.11 and 4.12 or if
the xenstored usage increased without realising?
I also can not find any list of "xenstored" commandline options to change
these settings on the xen-website or anywhere else. What do I need to put into
google to find this?

--
Joost

Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction [ In reply to ]

jgross at suse

Jun 4, 2020, 6:09 AM

Post #6 of 6 (192 views)

Permalink

On 04.06.20 14:00, J. Roeleveld wrote:
> On Wednesday, June 3, 2020 3:26:19 PM CEST Jürgen Groß wrote:
>> On 03.06.20 15:01, J. Roeleveld wrote:
>>> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
>
>>>> The default is 1000, you can set the value via the -E start parameter of
>>>> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
>>>> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
>>>> "quota-maxentity").
>>>
>>> Has this default been changed/lowered?
>>> Or has there been an increase in amount of values in xenstore?
>>
>> Depends. Which Xen version did you use before, and which Xenstore
>> variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
>> Did you change anything in the driver domain?
>
> My previous version was 4.11.2.
> Using xenstored.
> No changes done to the driver domain.
> - Issue occured initially when driver domain was still on 4.11.2
> - Issue still occured when driver domain was upgraded to 4.12.2
>
> In my opinion, putting a quota on the xenstored database at 1000kb is far too
> low. I am not doing anything special and have had no issues running over 20
> VMs simultaneously in the past.

The quota is for number of Xenstore entries per domain.

> Suddenly encountering this is unexpected and, if sane defaults are used,
> unnecessary.
>
> Do you happen to know if this limit was changed between 4.11 and 4.12 or if
> the xenstored usage increased without realising?

The default limits didn't change.

I could imagine that the number of Xenstore entries per device did
change. I'm not aware of any other changes which could explain the
different behavior.

> I also can not find any list of "xenstored" commandline options to change
> these settings on the xen-website or anywhere else. What do I need to put into
> google to find this?

xenstored --help

will list all options.

I should note that I have raised this issue in the community and we'll
discuss how to address it (e.g. by adding per-domain Xenstore quota via
the domain config).

Juergen