On Fri, Nov 2, 2018 at 9:12 PM Bernd Bausch <berndbausch@gmail.com> wrote:
> This is the part that determines the hypervisor:
>
>> # /etc/nova/nova-compute.conf
>> [DEFAULT]
>> compute_driver=libvirt.LibvirtDriver
>> [libvirt]
>> virt_type=xen
>>
>
> Nova uses libvirt to manage VMs, and libvirt uses Xen. According to the Nova/Xen
> configuration guide
> <https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-xen-libvirt.html>,
> it is enough to set these variables. However, if you have not read that
> config page, do it now. Are you using correct image formats? Do you find
> anything in the other logfiles mentioned on that page?
>
> The references to *qemu-img *in the nova-compute log don't mean that qemu
> is used. Nova uses *qemu-img* as a general tool for converting VM images,
> no matter which hypervisor is configured.
>
> However, something prevents Nova to access one of the disk images, namely
> /home/caslab/xenguest1/domains/guest1/disk.img. Is *guest1* the VM you
> started manually?
>
> I see:
>
> Unexpected error while running command.
> Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824
> --cpu=30 -- env LC_ALL=C LANG=C * qemu-img info
> /home/caslab/xenguest1/domains/guest1/disk.img*
>
> and
>
> Could not open '/home/caslab/xenguest1/domains/guest1/disk.img':
> Permission denied
>
> I don't know, though, whether these errors are related to your problem.
> Why is Libvirt or Xen unable to start your instances? I suggest you look
> for other errors in the nova-compute.log files.
>
> Bernd
>
> On 11/2/2018 6:30 PM, Minjun Hong wrote:
>
> On Fri, Nov 2, 2018 at 9:05 AM Bernd Bausch <berndbausch@gmail.com> wrote:
>
>> Welcome to OpenStack! I think the joy of troubleshooting it is one of its
>> main selling points.
>>
>> The conductor log says "*libxenlight failed to create new domain*" on
>> various nodes. You should check the nova-compute logs on those nodes for
>> relevant messages. I suspect an ill-configured interface between Nova
>> Compute and libvirt/Xen. In other words, double-check the libvirt and xen
>> sections of nova.conf on those nodes.
>>
>> Then there is this message "*Invalid **input for field
>> 'identity/password/user/password': None is not of type 'string' ....*".
>> This is often caused by incompatible software. I.e. you installed Openstack
>> components that don't fit together. Might also be caused by a configuration
>> error.
>> Or it's an ordinary bug, but it seems to occur on node 2 only.
>> On 11/1/2018 11:01 PM, Minjun Hong wrote:
>>
>> Hi everyone.
>> I'm really new to OpenStack.
>> After I install essential components of OpenStack, such as Nova,
>> Keystone, etc, I attempted creating an instance through Openstack command
>> in the terminal.
>> But a trouble has occurred. It was logged in "nova-conductor.log":
>>
>> 2018-11-01 22:18:59.831 2570 ERROR nova.scheduler.utils
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node2 (node
>>> node2): [.u'Traceback (most recent call last):\n', u' File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: Invalid input for
>>> field 'identity/password/user/password': None is not of type
>>> 'string'\n\nFailed validating 'type' in
>>> schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n
>>> {'type': 'string'}\n\nOn
>>> instance['identity']['password']['user']['password']:\n None (HTTP 400)
>>> (Request-ID: req-350beee4-2fed-4645-9e21-79806d7ebfe7)\n"]
>>> 2018-11-01 22:18:59.833 2570 WARNING oslo_config.cfg
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Option "os_region_name"
>>> from group "placement" is deprecated. Use option "region-name" from group
>>> "placement".
>>> 2018-11-01 22:19:10.936 2571 ERROR nova.scheduler.utils
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node4 (node
>>> node4): [.u'Traceback (most recent call last):\n', u' File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error:
>>> libxenlight failed to create new domain 'instance-00000005'\n"]
>>> 2018-11-01 22:19:21.783 2567 ERROR nova.scheduler.utils
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node5 (node
>>> node5): [.u'Traceback (most recent call last):\n', u' File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error:
>>> libxenlight failed to create new domain 'instance-00000005'\n"]
>>> 2018-11-01 22:19:21.783 2567 WARNING nova.scheduler.utils
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Failed to
>>> compute_task_build_instances: Exceeded maximum number of retries. Exceeded
>>> max scheduling attempts 3 for instance
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception: internal error:
>>> libxenlight failed to create new domain 'instance-00000005':
>>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max
>>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee.
>>> Last exception: internal error: libxenlight failed to create new domain
>>> 'instance-00000005'
>>> 2018-11-01 22:19:21.784 2567 WARNING nova.scheduler.utils
>>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Setting instance to ERROR state.:
>>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max
>>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee.
>>> Last exception: internal error: libxenlight failed to create new domain
>>> 'instance-00000005'
>>
>>
>> I'm not sure which component is involved in this trouble.
>> And libvirt and Xen have been successfully installed on all compute node
>> I have without any problem.
>>
>> nickeys@node2:~$ virsh create ./test.xml
>>> Domain guest1 created from ./test.xml
>>> nickeys@node2:~$ virsh list
>>> Id Name State
>>> --------------------------
>>> 0 Domain-0 running
>>> 1 guest1 running
>>> nickeys@node2:~$
>>
>>
>> What should I check first for that issue ?
>> Your hint would be big help for me.
>>
>> Thanks!
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>>
> Thanks for your kind answer, Bernd.
> I checked the configure files which you mentioned like it can cause the
> trouble. following files are my nova.conf and nova-compute.conf of the
> compute node ('node2').
>
> # /etc/nova/nova.conf
>
> [DEFAULT]
>> dhcpbridge_flagfile=/etc/nova/nova.conf
>> dhcpbridge=/usr/bin/nova-dhcpbridge
>> state_path=/var/lib/nova
>> lock_path=/var/lock/nova
>> force_dhcp_release=True
>> libvirt_use_virtio_for_bridges=True
>> verbose=True
>> ec2_private_dns_show_ip=True
>> api_paste_config=/etc/nova/api-paste.ini
>> enabled_apis=osapi_compute,metadata
>> transport_url = rabbit://admin:1234@10.150.3.88
>> my_ip = 10.150.21.182
>> use_neutron = True
>> firewall_driver = nova.virt.firewall.NoopFirewallDriver
>> [api]
>> auth_strategy = keystone
>> [vnc]
>> enabled = True
>> server_listen = 0.0.0.0
>> vncserver_proxyclient_address = 10.150.21.182
>> novncproxy_base_url = http://10.150.3.88:6080/vnc_auth.html
>> [glance]
>> api_servers = http://10.150.3.88:9292
>> [oslo_concurrency]
>> lock_path = /var/lib/nova/tmp
>> [keystone_authtoken]
>> auth_url = http://10.150.3.88:5000/v3
>> memcached_servers = 10.150.3.88:11211
>> auth_type = password
>> project_domain_name = default
>> user_domain_name = default
>> project_name = service
>> username = nova
>> password = <KEYSTONE_PASS>
>> [placement]
>> os_region_name = RegionOne
>> project_domain_name = Default
>> project_name = service
>> auth_type = password
>> user_domain_name = Default
>> auth_url = http://10.150.3.88:5000/v3
>> username = placement
>> password = <PLACEMENT_PASS>
>> [database]
>> connection=sqlite:////var/lib/nova/nova.sqlite
>> [api_database]
>> connection=sqlite:////var/lib/nova/nova.sqlite
>> [neutron]
>> url = http://10.150.3.88:9696
>> auth_url = http://10.150.3.88:5000
>> auth_type = password
>> project_domain_name = default
>> user_domain_name = default
>> region_name = RegionOne
>> project_name = service
>> username = neutron
>> passsword = <NEUTRON_PASS>
>
>
> # /etc/nova/nova-compute.conf
>> [DEFAULT]
>> compute_driver=libvirt.LibvirtDriver
>> [libvirt]
>> virt_type=xen
>
>
> And I set all the compute nodes I'm using with the same configure file
> above.
> Also, I installed the OpenStack components, which belong to Queens
> version, following the official installation guide (
> https://docs.openstack.org/install-guide/).
>
> By the way, I found something weird in '/var/log/nova/nova-compute.log' of
> the 'node2'.
> After I manually created a VM by 'virsh' command (i.g. $ virsh create
> test.xml), nova-compute of 'node2' started to complain.
> And on an other node which have not created a virtual machine by the
> 'virsh' command, there is not below error.
> I guess that the nova-compute component was configured to qemu as
> virtualization type, but I have no idea where the setting value is.
>
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
>> [req-ec324126-6dfc-4011-9f45-41d014e6f900 - - - - -] Error updating
>> resources for node node2.: InvalidDiskInfo: Disk info file is invalid:
>> qemu-img failed to execute on
>> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while
>> running command.
>> Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824
>> --cpu=30 -- env LC_ALL=C LANG=C qemu-img info
>> /home/caslab/xenguest1/domains/guest1/disk.img
>> Exit code: 1
>> Stdout: u''
>> Stderr: u"qemu-img: Could not open
>> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
>> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Traceback (most
>> recent call last):
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7284, in
>> update_available_resource_for_node
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
>> rt.update_available_resource(context, nodename)
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line
>> 664, in update_available_resource
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager resources =
>> self.driver.get_available_resource(nodename)
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6440,
>> in get_available_resource
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
>> disk_over_committed = self._get_disk_over_committed_size_total()
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8019,
>> in _get_disk_over_committed_size_total
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager config,
>> block_device_info)
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7918,
>> in _get_instance_disk_info_from_config
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager dk_size =
>> disk_api.get_allocated_disk_size(path)
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in
>> get_allocated_disk_size
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager return
>> images.qemu_img_info(path).disk_size
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
>> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in
>> qemu_img_info
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager raise
>> exception.InvalidDiskInfo(reason=msg)
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager InvalidDiskInfo:
>> Disk info file is invalid: qemu-img failed to execute on
>> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while
>> running command.
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Command:
>> /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 --
>> env LC_ALL=C LANG=C qemu-img info
>> /home/caslab/xenguest1/domains/guest1/disk.img
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Exit code: 1
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stdout: u''
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stderr:
>> u"qemu-img: Could not open
>> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
>> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
>> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
>
>
> Thanks!
>
>
>
I found a crucial clue.
When I tried to create an instance, every time, I got the error messages
repeated continuously.
It is from the log file of node2, /var/log/nova/nova-compute.log:
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
[req-f93b17df-6718-457a-9ce0-916b4e051619 cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default] Instance failed network
setup after 1 attempt(s): BadRequest: Invalid input for field
'identity/password/user/password': None is not of type 'string'
Failed validating 'type' in
schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
{'type': 'string'}
On instance['identity']['password']['user']['password']:
None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager Traceback (most
recent call last):
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1399, in
_allocate_network_async
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
bind_host_id=bind_host_id)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 954,
in allocate_for_instance
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager bind_host_id,
available_macs, requested_ports_dict)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1087, in _update_ports_for_instance
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager vif.destroy()
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
self.force_reraise()
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
six.reraise(self.type_, self.value, self.tb)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1057, in _update_ports_for_instance
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager port_client,
instance, port_id, port_req_body)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 468,
in _update_port
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager port_response
= port_client.update_port(port_id, port_req_body)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret =
obj(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 799,
in update_port
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
revision_number=revision_number)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret =
obj(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2375,
in _update_resource
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self.put(path, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret =
obj(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363,
in put
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
headers=headers, params=params)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret =
obj(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331,
in retry_request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
headers=headers, params=params)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret =
obj(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282,
in do_request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
headers=headers)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 343, in
do_request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self.request(url, method, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 331, in
request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager resp =
super(SessionClient, self).request(*args, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in
request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self.session.request(url, method, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in
request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager auth_headers =
self.get_auth_headers(auth)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in
get_auth_headers
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
auth.get_headers(self, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in
get_headers
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager token =
self.get_token(session)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88,
in get_token
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self.get_access(session).auth_token
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
134, in get_access
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager self.auth_ref
= self.get_auth_ref(session)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py",
line 201, in get_auth_ref
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self._plugin.get_auth_ref(session, **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line
177, in get_auth_ref
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
authenticated=False, log=False, **rkwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in
post
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return
self.request(url, 'POST', **kwargs)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in
request
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager raise
exceptions.from_response(resp, method, url)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager BadRequest:
Invalid input for field 'identity/password/user/password': None is not of
type 'string'
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager Failed validating
'type' in
schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager {'type':
'string'}
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager On
instance['identity']['password']['user']['password']:
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager None (HTTP
400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc)
2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager
[req-f93b17df-6718-457a-9ce0-916b4e051619 cc22ec575cb44e53aced9ddf58d9e8d7
965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Instance failed to spawn: BadRequest:
Invalid input for field 'identity/password/user/password': None is not of
type 'string'
Failed validating 'type' in
schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
{'type': 'string'}
On instance['identity']['password']['user']['password']:
None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Traceback (most recent call last):
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2251, in
_build_resources
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] yield resources
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2031, in
_build_and_run_instance
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762]
block_device_info=block_device_info)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3084,
in spawn
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] mdevs=mdevs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5368,
in _get_guest_xml
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] network_info_str =
str(network_info)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/model.py", line 568, in
__str__
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self._sync_wrapper(fn,
*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/model.py", line 551, in
_sync_wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.wait()
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/model.py", line 583, in wait
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self[:] = self._gt.wait()
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in
wait
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self._exit_event.wait()
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] current.throw(*self._exc)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in
main
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] result = function(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 906, in
context_wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return func(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1416, in
_allocate_network_async
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] six.reraise(*exc_info)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1399, in
_allocate_network_async
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] bind_host_id=bind_host_id)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 954,
in allocate_for_instance
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] bind_host_id, available_macs,
requested_ports_dict)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1087, in _update_ports_for_instance
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] vif.destroy()
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.force_reraise()
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] six.reraise(self.type_,
self.value, self.tb)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1057, in _update_ports_for_instance
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] port_client, instance, port_id,
port_req_body)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 468,
in _update_port
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] port_response =
port_client.update_port(port_id, port_req_body)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 799,
in update_port
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] revision_number=revision_number)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2375,
in _update_resource
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.put(path, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363,
in put
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] headers=headers, params=params)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114,
in wrapper
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282,
in do_request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] headers=headers)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 343, in
do_request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.request(url, method,
**kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 331, in
request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] resp = super(SessionClient,
self).request(*args, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in
request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.session.request(url,
method, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in
request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] auth_headers =
self.get_auth_headers(auth)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in
get_auth_headers
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return auth.get_headers(self,
**kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in
get_headers
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] token = self.get_token(session)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88,
in get_token
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return
self.get_access(session).auth_token
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
134, in get_access
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.auth_ref =
self.get_auth_ref(session)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py",
line 201, in get_auth_ref
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return
self._plugin.get_auth_ref(session, **kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line
177, in get_auth_ref
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] authenticated=False, log=False,
**rkwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in
post
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.request(url, 'POST',
**kwargs)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File
"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in
request
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] raise
exceptions.from_response(resp, method, url)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] BadRequest: Invalid input for field
'identity/password/user/password': None is not of type 'string'
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762]
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Failed validating 'type' in
schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] {'type': 'string'}
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762]
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] On
instance['identity']['password']['user']['password']:
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] None (HTTP 400) (Request-ID:
req-c9fac551-ebb4-4b2c-af67-5bad51d403fc)
2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance:
2ef5e55c-4868-4ff1-bbc9-3d82c4f99762]
Above logs all occurred with only one instance creation.
Although I already searched the internet, I found nothing meaningful.
Just, I'm guessing that when I requested an instance creation on the
controller node, it delivered the request with an empty parameter, which
was deemed to None type. I mean, the empty one is password.
However, I'm confused because I have set all the config files which the
official documentation guided to add.
As I think, the main issue is the reason the instance creation request was
sent to 'node2' without password parameter.
Have you ever seen the error message I had?
