Mailing List Archive

On 08/07/2018 10:57 AM, Cody wrote:
> Hi everyone,
>
> I intentionally triggered an error by launching more instances than it
> is allowed by the 'cpu_allocation_ratio' set on a compute node. When it
> comes to logs, the only place contained a clue to explain the launch
> failure was in the nova-conductor.log on a controller node. Why there is
> no trace in the nova-scheduler.log (or any other logs) for this type or
> errors?

Because it's not an error.

You exceeded the capacity of your resources, that's all.

Are you asking why there isn't a way to *check* to see whether a
particular request to launch a VM (or multiple VMs) will exceed the
capacity of your deployment?

Best,
-jay

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Hi Jay,

Thank you for getting back to my question.

I agree that it is not an error; only a preset limit is reached. I
just wonder why this incident only got recorded in the
nova-conductor.log, but not in other files such as nova-scheduler.log,
which would make more sense to me. :-)

By the way, I am using the Queens release.

Regards,
Cody
On Tue, Aug 7, 2018 at 11:38 AM Jay Pipes <jaypipes@gmail.com> wrote:
>
> On 08/07/2018 10:57 AM, Cody wrote:
> > Hi everyone,
> >
> > I intentionally triggered an error by launching more instances than it
> > is allowed by the 'cpu_allocation_ratio' set on a compute node. When it
> > comes to logs, the only place contained a clue to explain the launch
> > failure was in the nova-conductor.log on a controller node. Why there is
> > no trace in the nova-scheduler.log (or any other logs) for this type or
> > errors?
>
> Because it's not an error.
>
> You exceeded the capacity of your resources, that's all.
>
> Are you asking why there isn't a way to *check* to see whether a
> particular request to launch a VM (or multiple VMs) will exceed the
> capacity of your deployment?
>
> Best,
> -jay
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

I would think you don't even reach the scheduling stage. Why bother
looking for a suitable compute node if you exceeded your quota anyway?

The message is in the conductor log because it's the conductor that does
most of the work. The others are just slackers (like nova-api) or wait
for instructions from the conductor.

The above is my guess, of course, but IMHO a very educated one.

Bernd.

On 8/8/2018 1:35 AM, Cody wrote:
> Hi Jay,
>
> Thank you for getting back to my question.
>
> I agree that it is not an error; only a preset limit is reached. I
> just wonder why this incident only got recorded in the
> nova-conductor.log, but not in other files such as nova-scheduler.log,
> which would make more sense to me. :-)
>
> By the way, I am using the Queens release.
>
> Regards,

On 08/08/2018 07:19 AM, Bernd Bausch wrote:
> I would think you don't even reach the scheduling stage. Why bother
> looking for a suitable compute node if you exceeded your quota anyway?
>
> The message is in the conductor log because it's the conductor that does
> most of the work. The others are just slackers (like nova-api) or wait
> for instructions from the conductor.
>
> The above is my guess, of course, but IMHO a very educated one.
>
> Bernd.
>
> On 8/8/2018 1:35 AM, Cody wrote:
>> Hi Jay,
>>
>> Thank you for getting back to my question.
>>
>> I agree that it is not an error; only a preset limit is reached. I
>> just wonder why this incident only got recorded in the
>> nova-conductor.log, but not in other files such as nova-scheduler.log,
>> which would make more sense to me. :-)

I gave up trying to answer this because the original poster did not
include any information about an "error" in either the original post [1]
or his reply.

So I have no idea what got recorded in the nova-conductor log at all.

Until I get some details I have no idea how to further answer the
question (or even if there *is* a question still?).

[1] http://lists.openstack.org/pipermail/openstack/2018-August/046804.html

>> By the way, I am using the Queens release.
>>
>> Regards,
>
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

> On 08/08/2018 07:19 AM, Bernd Bausch wrote:
> > I would think you don't even reach the scheduling stage. Why bother
> > looking for a suitable compute node if you exceeded your quota anyway?
> >
> > The message is in the conductor log because it's the conductor that does
> > most of the work. The others are just slackers (like nova-api) or wait
> > for instructions from the conductor.
> >
> > The above is my guess, of course, but IMHO a very educated one.
> >
> > Bernd.

Thank you, Bernd. I didn't know the inner workflow in this case.
Initially, I thought it was for the scheduler to discover that no more
resource was left available, hence I expected to see something from
the scheduler log. My understanding now is that the quota get checked
in the database prior to the deployment. That would explain why the
clue was in the nova-conductor.log, not the nova-scheduler.log.

Cody


> I gave up trying to answer this because the original poster did not
> include any information about an "error" in either the original post [1]
> or his reply.
>
> So I have no idea what got recorded in the nova-conductor log at all.
>
> Until I get some details I have no idea how to further answer the
> question (or even if there *is* a question still?).
>
> [1] http://lists.openstack.org/pipermail/openstack/2018-August/046804.html

Hi Jay,

My apologies for omitting the log information. I am attaching it below
for the record. Hope the format won't get too messy...

From the nova-conductor.log
### BEGIN ###
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] Failed to schedule instances: NoValidHost_Remote: No
valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 226, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 139, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
(most recent call last):
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
1116, in schedule_and_build_instances
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_alternates=True)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
716, in _schedule_instances
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
return_alternates=return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 726,
in wrapped
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
func(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
line 53, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_objects, return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
line 37, in __run_method
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
getattr(self.instance, __name)(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py",
line 42, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_objects, return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158,
in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
cctxt.call(ctxt, 'select_destinations', **msg_args)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
174, in call
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=self.retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
131, in _send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
timeout=timeout, retry=retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 550, in _send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise result
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
NoValidHost_Remote: No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
(most recent call last):
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
226, in inner
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
func(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line
139, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise
exception.NoValidHost(reason="")
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager NoValidHost:
No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:36.328 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] Failed to compute_task_build_instances: No valid host
was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 226, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 139, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
: NoValidHost_Remote: No valid host was found.
2018-08-08 09:28:36.331 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] [instance: b466a974-06ba-459b-bc04-2ccb2b3ee720]
Setting instance to ERROR state.: NoValidHost_Remote: No valid host
was found.
### END ###

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

On 08/08/2018 09:37 AM, Cody wrote:
>> On 08/08/2018 07:19 AM, Bernd Bausch wrote:
>>> I would think you don't even reach the scheduling stage. Why bother
>>> looking for a suitable compute node if you exceeded your quota anyway?
>>>
>>> The message is in the conductor log because it's the conductor that does
>>> most of the work. The others are just slackers (like nova-api) or wait
>>> for instructions from the conductor.
>>>
>>> The above is my guess, of course, but IMHO a very educated one.
>>>
>>> Bernd.
>
> Thank you, Bernd. I didn't know the inner workflow in this case.
> Initially, I thought it was for the scheduler to discover that no more
> resource was left available, hence I expected to see something from
> the scheduler log. My understanding now is that the quota get checked
> in the database prior to the deployment. That would explain why the
> clue was in the nova-conductor.log, not the nova-scheduler.log.

Quota is checked in the nova-api node, not the nova-conductor.

As I said in my previous message, unless you paste what the logs are
that you are referring to, it's not possible to know what you are
referring to.

Best,
-jay

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Hi Jay,

Thank you for getting back. I attached the log in my previous reply,
but I guess Gmail hided it from you as a quoted message. Here comes
again:

From nova-conductor.log
### BEGIN ###
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] Failed to schedule instances: NoValidHost_Remote: No
valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 226, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 139, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
(most recent call last):
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
1116, in schedule_and_build_instances
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_alternates=True)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
716, in _schedule_instances
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
return_alternates=return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 726,
in wrapped
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
func(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
line 53, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_objects, return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
line 37, in __run_method
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
getattr(self.instance, __name)(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py",
line 42, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
instance_uuids, return_objects, return_alternates)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158,
in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
cctxt.call(ctxt, 'select_destinations', **msg_args)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
174, in call
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=self.retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
131, in _send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
timeout=timeout, retry=retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=retry)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 550, in _send
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise result
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
NoValidHost_Remote: No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
(most recent call last):
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
226, in inner
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
func(*args, **kwargs)
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
"/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line
139, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise
exception.NoValidHost(reason="")
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager NoValidHost:
No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:36.328 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] Failed to compute_task_build_instances: No valid host
was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 226, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 139, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
: NoValidHost_Remote: No valid host was found.
2018-08-08 09:28:36.331 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] [instance: b466a974-06ba-459b-bc04-2ccb2b3ee720]
Setting instance to ERROR state.: NoValidHost_Remote: No valid host
was found.
### END ###
On Wed, Aug 8, 2018 at 9:45 AM Jay Pipes <jaypipes@gmail.com> wrote:
>
> On 08/08/2018 09:37 AM, Cody wrote:
> >> On 08/08/2018 07:19 AM, Bernd Bausch wrote:
> >>> I would think you don't even reach the scheduling stage. Why bother
> >>> looking for a suitable compute node if you exceeded your quota anyway?
> >>>
> >>> The message is in the conductor log because it's the conductor that does
> >>> most of the work. The others are just slackers (like nova-api) or wait
> >>> for instructions from the conductor.
> >>>
> >>> The above is my guess, of course, but IMHO a very educated one.
> >>>
> >>> Bernd.
> >
> > Thank you, Bernd. I didn't know the inner workflow in this case.
> > Initially, I thought it was for the scheduler to discover that no more
> > resource was left available, hence I expected to see something from
> > the scheduler log. My understanding now is that the quota get checked
> > in the database prior to the deployment. That would explain why the
> > clue was in the nova-conductor.log, not the nova-scheduler.log.
>
> Quota is checked in the nova-api node, not the nova-conductor.
>
> As I said in my previous message, unless you paste what the logs are
> that you are referring to, it's not possible to know what you are
> referring to.
>
> Best,
> -jay

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

So, that is normal operation, actually. The conductor calls the
scheduler to find a place for your requested instances. The scheduler
responded to the conductor that, sorry, there were no hosts that were
able to match the request (I don't know what the details of that request
were).

And so the conductor set the status of the instance(s) in your request
to an ERROR state, since they were not able to be launched.

Best,
-jay

On 08/08/2018 09:58 AM, Cody wrote:
> Hi Jay,
>
> Thank you for getting back. I attached the log in my previous reply,
> but I guess Gmail hided it from you as a quoted message. Here comes
> again:
>
> From nova-conductor.log
> ### BEGIN ###
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> default default] Failed to schedule instances: NoValidHost_Remote: No
> valid host was found.
> Traceback (most recent call last):
>
> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> line 226, in inner
> return func(*args, **kwargs)
>
> File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
> line 139, in select_destinations
> raise exception.NoValidHost(reason="")
>
> NoValidHost: No valid host was found.
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
> (most recent call last):
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
> 1116, in schedule_and_build_instances
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> instance_uuids, return_alternates=True)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
> 716, in _schedule_instances
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> return_alternates=return_alternates)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 726,
> in wrapped
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> func(*args, **kwargs)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
> line 53, in select_destinations
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> instance_uuids, return_objects, return_alternates)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
> line 37, in __run_method
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> getattr(self.instance, __name)(*args, **kwargs)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py",
> line 42, in select_destinations
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> instance_uuids, return_objects, return_alternates)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158,
> in select_destinations
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> cctxt.call(ctxt, 'select_destinations', **msg_args)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
> 174, in call
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=self.retry)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
> 131, in _send
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> timeout=timeout, retry=retry)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 559, in send
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=retry)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 550, in _send
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise result
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> NoValidHost_Remote: No valid host was found.
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
> (most recent call last):
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
> 226, in inner
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> func(*args, **kwargs)
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line
> 139, in select_destinations
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise
> exception.NoValidHost(reason="")
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager NoValidHost:
> No valid host was found.
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> 2018-08-08 09:28:36.328 1648 WARNING nova.scheduler.utils
> [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> default default] Failed to compute_task_build_instances: No valid host
> was found.
> Traceback (most recent call last):
>
> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> line 226, in inner
> return func(*args, **kwargs)
>
> File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
> line 139, in select_destinations
> raise exception.NoValidHost(reason="")
>
> NoValidHost: No valid host was found.
> : NoValidHost_Remote: No valid host was found.
> 2018-08-08 09:28:36.331 1648 WARNING nova.scheduler.utils
> [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> default default] [instance: b466a974-06ba-459b-bc04-2ccb2b3ee720]
> Setting instance to ERROR state.: NoValidHost_Remote: No valid host
> was found.
> ### END ###
> On Wed, Aug 8, 2018 at 9:45 AM Jay Pipes <jaypipes@gmail.com> wrote:
>>
>> On 08/08/2018 09:37 AM, Cody wrote:
>>>> On 08/08/2018 07:19 AM, Bernd Bausch wrote:
>>>>> I would think you don't even reach the scheduling stage. Why bother
>>>>> looking for a suitable compute node if you exceeded your quota anyway?
>>>>>
>>>>> The message is in the conductor log because it's the conductor that does
>>>>> most of the work. The others are just slackers (like nova-api) or wait
>>>>> for instructions from the conductor.
>>>>>
>>>>> The above is my guess, of course, but IMHO a very educated one.
>>>>>
>>>>> Bernd.
>>>
>>> Thank you, Bernd. I didn't know the inner workflow in this case.
>>> Initially, I thought it was for the scheduler to discover that no more
>>> resource was left available, hence I expected to see something from
>>> the scheduler log. My understanding now is that the quota get checked
>>> in the database prior to the deployment. That would explain why the
>>> clue was in the nova-conductor.log, not the nova-scheduler.log.
>>
>> Quota is checked in the nova-api node, not the nova-conductor.
>>
>> As I said in my previous message, unless you paste what the logs are
>> that you are referring to, it's not possible to know what you are
>> referring to.
>>
>> Best,
>> -jay

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Got it! Thank you, Jay! - Cody
On Wed, Aug 8, 2018 at 11:36 AM Jay Pipes <jaypipes@gmail.com> wrote:
>
> So, that is normal operation, actually. The conductor calls the
> scheduler to find a place for your requested instances. The scheduler
> responded to the conductor that, sorry, there were no hosts that were
> able to match the request (I don't know what the details of that request
> were).
>
> And so the conductor set the status of the instance(s) in your request
> to an ERROR state, since they were not able to be launched.
>
> Best,
> -jay
>
> On 08/08/2018 09:58 AM, Cody wrote:
> > Hi Jay,
> >
> > Thank you for getting back. I attached the log in my previous reply,
> > but I guess Gmail hided it from you as a quoted message. Here comes
> > again:
> >
> > From nova-conductor.log
> > ### BEGIN ###
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> > 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> > default default] Failed to schedule instances: NoValidHost_Remote: No
> > valid host was found.
> > Traceback (most recent call last):
> >
> > File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> > line 226, in inner
> > return func(*args, **kwargs)
> >
> > File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
> > line 139, in select_destinations
> > raise exception.NoValidHost(reason="")
> >
> > NoValidHost: No valid host was found.
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
> > (most recent call last):
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
> > 1116, in schedule_and_build_instances
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > instance_uuids, return_alternates=True)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
> > 716, in _schedule_instances
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > return_alternates=return_alternates)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 726,
> > in wrapped
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> > func(*args, **kwargs)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
> > line 53, in select_destinations
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > instance_uuids, return_objects, return_alternates)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py",
> > line 37, in __run_method
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> > getattr(self.instance, __name)(*args, **kwargs)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py",
> > line 42, in select_destinations
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > instance_uuids, return_objects, return_alternates)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158,
> > in select_destinations
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> > cctxt.call(ctxt, 'select_destinations', **msg_args)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
> > 174, in call
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=self.retry)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
> > 131, in _send
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > timeout=timeout, retry=retry)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> > line 559, in send
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager retry=retry)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> > line 550, in _send
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise result
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > NoValidHost_Remote: No valid host was found.
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager Traceback
> > (most recent call last):
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
> > 226, in inner
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager return
> > func(*args, **kwargs)
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line
> > 139, in select_destinations
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise
> > exception.NoValidHost(reason="")
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager NoValidHost:
> > No valid host was found.
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > 2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
> > 2018-08-08 09:28:36.328 1648 WARNING nova.scheduler.utils
> > [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> > 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> > default default] Failed to compute_task_build_instances: No valid host
> > was found.
> > Traceback (most recent call last):
> >
> > File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> > line 226, in inner
> > return func(*args, **kwargs)
> >
> > File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
> > line 139, in select_destinations
> > raise exception.NoValidHost(reason="")
> >
> > NoValidHost: No valid host was found.
> > : NoValidHost_Remote: No valid host was found.
> > 2018-08-08 09:28:36.331 1648 WARNING nova.scheduler.utils
> > [req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
> > 2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
> > default default] [instance: b466a974-06ba-459b-bc04-2ccb2b3ee720]
> > Setting instance to ERROR state.: NoValidHost_Remote: No valid host
> > was found.
> > ### END ###
> > On Wed, Aug 8, 2018 at 9:45 AM Jay Pipes <jaypipes@gmail.com> wrote:
> >>
> >> On 08/08/2018 09:37 AM, Cody wrote:
> >>>> On 08/08/2018 07:19 AM, Bernd Bausch wrote:
> >>>>> I would think you don't even reach the scheduling stage. Why bother
> >>>>> looking for a suitable compute node if you exceeded your quota anyway?
> >>>>>
> >>>>> The message is in the conductor log because it's the conductor that does
> >>>>> most of the work. The others are just slackers (like nova-api) or wait
> >>>>> for instructions from the conductor.
> >>>>>
> >>>>> The above is my guess, of course, but IMHO a very educated one.
> >>>>>
> >>>>> Bernd.
> >>>
> >>> Thank you, Bernd. I didn't know the inner workflow in this case.
> >>> Initially, I thought it was for the scheduler to discover that no more
> >>> resource was left available, hence I expected to see something from
> >>> the scheduler log. My understanding now is that the quota get checked
> >>> in the database prior to the deployment. That would explain why the
> >>> clue was in the nova-conductor.log, not the nova-scheduler.log.
> >>
> >> Quota is checked in the nova-api node, not the nova-conductor.
> >>
> >> As I said in my previous message, unless you paste what the logs are
> >> that you are referring to, it's not possible to know what you are
> >> referring to.
> >>
> >> Best,
> >> -jay

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack