Mailing List Archive

Hi all,

I've been struggling with instances suddenly not being able to fetch
metadata from Openstack Queens (this has worked fine earlier).

Newly created VMs fail to connect to the magic ip, eg.
http://169.254.169.254/, and won't initialize properly. Subsequently ssh
login will fail since no key is uploaded.

The symptom is failed requests in the log

*Cirros:*
Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending select for 10.0.0.18...
Lease of 10.0.0.18 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.0.0.1"
cirros-ds 'net' up at 0.94
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 0.94. request failed
failed 2/20: up 3.01. request failed
failed 3/20: up 5.03. request failed
failed 4/20: up 7.04. request failed

*..and on Centos6:*
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | 0 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH |
ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
2018-11-06 08:10:07,892 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:08,906 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:09,925 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: unexpected error [.'NoneType' object has no attribute
...

Using Curl manually, eg. '/curl http://169.254.169.254/openstack/' one
gets:

/curl: (52) Empty reply from server/

*At the same time this error is showing up in the syslog on the controller:*

Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 460,
in fire_timers
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: timer()
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line
59, in __call__
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: cb(*args, **kw)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line
219, in main
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: result =
function(*args, **kwargs)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File
"/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 793, in
process_request
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:
proto.__init__(conn_state, self)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: TypeError:
__init__() takes exactly 4 arguments (3 given)

*Neither rebooting the controller, reinstalling neutron, or restarting
the services will do anything top fix this.*

Has anyone else seen this? We are using Queens with a single controller.

Kind Regards

Terje Lundin