Mailing List Archive

[OCTAVIA][KOLLA] - Self signed CA/CERTS
Hi guys,

I continue to work on my Octavia integration using Kolla-Ansible and I'm
facing a strange behavior.

As for now I'm working on a POC using restricted HW and SW Capacities, I'm
facing a strange issue when trying to launch a new load-balancer.

When I create a new LB, would it be using CLI or WebUI, the amphora
immediately disappear and the LB status switch to ERROR.

When looking at logs and especially Worker logs, I see that the error seems
to be related to the fact that the worker can't connect to the amphora
because of a TLS Handshake issue which so trigger the contact timeout and
rollback the amphora creation.

Here is the worker.log relevant trace:










































































































*2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-]
Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...2018-08-07
07:33:57.220 24 INFO octavia.controller.worker.tasks.database_tasks [-]
Created Amphora in DB with id
c20af002-1576-446e-b99f-7af607b8d8852018-08-07 07:33:57.285 24 INFO
octavia.certificates.generator.local [-] Signing a certificate request
using OpenSSL locally.2018-08-07 07:33:57.285 24 INFO
octavia.certificates.generator.local [-] Using CA Certificate from
config.2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
[-] Using CA Private Key from config.2018-08-07 07:33:57.286 24 INFO
octavia.certificates.generator.local [-] Using CA Private Key Passphrase
from config.2018-08-07 07:34:04.074 24 INFO
octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for
amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id
3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer:
bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e2018-08-07 07:34:04.253 24 INFO
octavia.network.drivers.neutron.allowed_address_pairs [-] Port
a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be
done.2018-08-07 07:34:19.656 24 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
instance. Retrying.: ConnectTimeout:
HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded
with url: /0.5/plug/vip/192.168.56.100 <http://192.168.56.100> (Caused by
ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection
object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect
timeout=10.0)'))2018-08-07 07:34:24.673 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
(c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE'
from state 'RUNNING'34 predecessors (most recent first): Atom
'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
<octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>},
'provides': {u'c20af002-1576-446e-b99f-7af607b8d885':
<octavia.network.data_models.AmphoraNetworkConfig object at
0x7f4c284786d0>}} |__Atom 'reload-lb-after-plug-vip' {'intention':
'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id':
u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
<octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
|__Atom
'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
[<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]},
'provides': None} |__Atom
'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention':
'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
[<octavia.common.data_models.Amphora object at 0x7f4c285165d0>],
'loadbalancer': <octavia.common.data_models.LoadBalancer object at
0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides':
None} |__Atom
'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention':
'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
<octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>},
'provides': [<octavia.common.data_models.Amphora object at
0x7f4c285165d0>]} |__Atom
'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip':
<octavia.common.data_models.Vip object at 0x7f4c284956d0>,
'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
<octavia.common.data_models.LoadBalancer object at
0x7f4c2845fe10>} |__Atom
'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention':
'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
<octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>},
'provides': <octavia.common.data_models.Vip object at
0x7f4c284956d0>} |__Flow
'octavia-new-loadbalancer-net-subflow' |__Atom
'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
{'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
<octavia.common.data_models.LoadBalancer object at
0x7f4c28478110>} |__Flow
'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
|__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
<octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
<octavia.common.data_models.Amphora object at
0x7f4c28478190>} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
<octavia.common.data_models.Amphora object at 0x7f4c28478a50>,
'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
<octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj':
<octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides':
<octavia.common.data_models.Amphora object at
0x7f4c28478a50>} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
<octavia.common.data_models.Amphora object at
0x7f4c2845fa10>} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
'-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885',
'build_type_priority': 40}, 'provides':
u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
| |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
'-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
None} | |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem'
{'intention': 'EXECUTE', 'state': 'SUCCESS'}
| |__Atom
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides':
u'c20af002-1576-446e-b99f-7af607b8d885'}
| |__Flow
'STANDALONE-octavia-create-amp-for-lb-subflow'
| |__Atom
'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
{'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
None} |
|__Flow
'STANDALONE-octavia-get-amphora-for-lb-subflow'
| |__Atom
'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
{'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
None}
| |__Flow
'octavia-create-loadbalancer-flow' |__Atom
'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb'
{'intention': 'IGNORE', 'state': 'IGNORE'}
|__Atom
'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora'
{'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id':
None}} |__Flow
'STANDALONE-octavia-post-map-amp-to-lb-subflow'
|__Atom
'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
{'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'},
'provides':None} |__Flow
'STANDALONE-octavia-get-amphora-for-lb-subflow'
|__Atom
'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
{'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
{'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
None} |__Flow
'octavia-create-loadbalancer-flow': Error: [.('PEM routines',
'PEM_read_bio', 'no start line'), ('SSL routines',
'SSL_CTX_use_certificate_file', 'PEM lib')]2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker Traceback (most recent call
last):2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
line 53, in _execute_task2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker result =
task.execute(**arguments)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
line 240, in execute2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
amphorae_network_config)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
line 219, in execute2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker amphora, loadbalancer,
amphorae_network_config)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
line 137, in post_vip_plug2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker net_info)2018-08-07
07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
line 388, in plug_vip2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker json=net_info)2018-08-07
07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
line 277, in request2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker r =
_request(**reqargs)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in
post2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker return self.request('POST',
url, data=data, json=json, **kwargs)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in
request2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker resp = self.send(prep,
**send_kwargs)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in
send2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker r = adapter.send(request,
**kwargs)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in
send2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker timeout=timeout2018-08-07
07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
line 600, in urlopen2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker chunked=chunked)2018-08-07
07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
line 345, in _make_request2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
self._validate_conn(conn)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
line 844, in _validate_conn2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker conn.connect()2018-08-07
07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py",
line 326, in connect2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
ssl_context=context)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py",
line 323, in ssl_wrap_socket2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
context.load_cert_chain(certfile, keyfile)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py",
line 418, in load_cert_chain2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
self._ctx.use_certificate_file(certfile)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in
use_certificate_file2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker
_raise_current_error()2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker File
"/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in
exception_from_error_queue2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker raise
exception_type(errors)2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker Error: [.('PEM routines',
'PEM_read_bio', 'no start line'), ('SSL routines',
'SSL_CTX_use_certificate_file', 'PEM lib')]2018-08-07 07:34:24.673 24 ERROR
octavia.controller.worker.controller_worker2018-08-07 07:34:24.684 24
WARNING octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
(c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:24.687 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
(1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:24.691 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f)
transitioned into state 'REVERTED' from state 'REVERTING'2018-08-07
07:34:24.694 24 WARNING octavia.controller.worker.controller_worker [-]
Task 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
(761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:24.716 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.network_tasks.ApplyQos'
(fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:24.719 24 WARNING
octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for
loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e2018-08-07 07:34:26.413
24 WARNING octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.network_tasks.PlugVIP'
(ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:26.420 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
(79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:26.425 24 WARNING
octavia.controller.worker.tasks.network_tasks [-] Deallocating vip
192.168.56.1002018-08-07 07:34:26.577 24 INFO
octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security
group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port
a7bae53e-0bc6-4830-8c75-646a8baf28852018-08-07 07:34:27.187 24 INFO
octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security
group 3d84ee39-1db9-475f-b048-9fe0f87201c12018-08-07 07:34:27.803 24
WARNING octavia.controller.worker.controller_worker [-] Task
'octavia.controller.worker.tasks.network_tasks.AllocateVIP'
(7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:27.807 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
(64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:27.810 24 WARNING
octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role
in DB for amp id c20af002-1576-446e-b99f-7af607b8d8852018-08-07
07:34:27.816 24 WARNING octavia.controller.worker.controller_worker [-]
Task
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
(2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:27.819 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
(86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:27.821 24 WARNING
octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora
ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id
3bbabfa6-366f-46a4-8fb2-1ec7158e19f12018-08-07 07:34:27.826 24 WARNING
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
(baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED'
from state 'REVERTING'2018-08-07 07:34:27.828 24 WARNING
octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora
finalize.*
Is this a problem if I use self-signed CAcert ?
Is their a way to tell octavia to ignore SSL Error while working on a LAB
environment?

As usual, if you need further information feel free to ask.

Thanks a lot guys.
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
Hi there Flint.

Octavia fully supports using self-signed certificates and we use those
in our gate tests.
We do not allow non-TLS authenticated connections in the code, even
for lab setups.

This is a configuration issue or certificate file format issue. When
the controller is attempting to access the controller local
certificate file (likely the one we use to prove we are a valid
controller to the amphora agent) it is finding a file without the
required PEM format header. Check that your certificate files have the
"-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
format and just need to be converted).

Also for reference, here are the minimal steps we use in our gate
tests to setup the TLS certificates:
https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305

Michael
On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com> wrote:
>
>
> Hi guys,
>
> I continue to work on my Octavia integration using Kolla-Ansible and I'm facing a strange behavior.
>
> As for now I'm working on a POC using restricted HW and SW Capacities, I'm facing a strange issue when trying to launch a new load-balancer.
>
> When I create a new LB, would it be using CLI or WebUI, the amphora immediately disappear and the LB status switch to ERROR.
>
> When looking at logs and especially Worker logs, I see that the error seems to be related to the fact that the worker can't connect to the amphora because of a TLS Handshake issue which so trigger the contact timeout and rollback the amphora creation.
>
> Here is the worker.log relevant trace:
>
> 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-] Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
> 2018-08-07 07:33:57.220 24 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id c20af002-1576-446e-b99f-7af607b8d885
> 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
> 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
> 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
> 2018-08-07 07:33:57.286 24 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
> 2018-08-07 07:34:04.074 24 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer: bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> 2018-08-07 07:34:04.253 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
> 2018-08-07 07:34:19.656 24 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.56.100 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect timeout=10.0)'))
> 2018-08-07 07:34:24.673 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug' (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE' from state 'RUNNING'
> 34 predecessors (most recent first):
> Atom 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}, 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885': <octavia.network.data_models.AmphoraNetworkConfig object at 0x7f4c284786d0>}}
> |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
> |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}, 'provides': None}
> |__Atom 'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
> |__Atom 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}, 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
> |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip': <octavia.common.data_models.Vip object at 0x7f4c284956d0>, 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
> |__Atom 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}, 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
> |__Flow 'octavia-new-loadbalancer-net-subflow'
> |__Atom 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
> |__Flow 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
> |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>, 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj': <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem': '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n', 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'build_type_priority': 40}, 'provides': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem': '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n', 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' {'intention': 'EXECUTE', 'state': 'SUCCESS'}
> | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': u'c20af002-1576-446e-b99f-7af607b8d885'}
> | |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
> | |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
> | |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> | |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
> | |__Flow 'octavia-create-loadbalancer-flow'
> |__Atom 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb' {'intention': 'IGNORE', 'state': 'IGNORE'}
> |__Atom 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora' {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
> |__Flow 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
> |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
> |__Flow 'octavia-create-loadbalancer-flow': Error: [.('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 240, in execute
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker amphorae_network_config)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 219, in execute
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker amphora, loadbalancer, amphorae_network_config)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 137, in post_vip_plug
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker net_info)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 388, in plug_vip
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker json=net_info)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 277, in request
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker r = _request(**reqargs)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker return self.request('POST', url, data=data, json=json, **kwargs)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker resp = self.send(prep, **send_kwargs)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker r = adapter.send(request, **kwargs)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker timeout=timeout
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker chunked=chunked)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker self._validate_conn(conn)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker conn.connect()
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 326, in connect
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker ssl_context=context)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 323, in ssl_wrap_socket
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker context.load_cert_chain(certfile, keyfile)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 418, in load_cert_chain
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker self._ctx.use_certificate_file(certfile)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in use_certificate_file
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker _raise_current_error()
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker raise exception_type(errors)
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker Error: [.('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
> 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker
> 2018-08-07 07:34:24.684 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug' (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:24.687 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:24.691 24 WARNING octavia.controller.worker.controller_worker [-] Task 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:24.694 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData' (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:24.716 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.ApplyQos' (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:24.719 24 WARNING octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> 2018-08-07 07:34:26.413 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.PlugVIP' (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:26.420 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:26.425 24 WARNING octavia.controller.worker.tasks.network_tasks [-] Deallocating vip 192.168.56.100
> 2018-08-07 07:34:26.577 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port a7bae53e-0bc6-4830-8c75-646a8baf2885
> 2018-08-07 07:34:27.187 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security group 3d84ee39-1db9-475f-b048-9fe0f87201c1
> 2018-08-07 07:34:27.803 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:27.807 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc' (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:27.810 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
> 2018-08-07 07:34:27.816 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb' (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:27.819 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora' (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:27.821 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
> 2018-08-07 07:34:27.826 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb' (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED' from state 'REVERTING'
> 2018-08-07 07:34:27.828 24 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora finalize.
>
> Is this a problem if I use self-signed CAcert ?
> Is their a way to tell octavia to ignore SSL Error while working on a LAB environment?
>
> As usual, if you need further information feel free to ask.
>
> Thanks a lot guys.
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
Hi Michael, thanks a lot for your quick response once again!
Le mar. 14 août 2018 à 18:21, Michael Johnson <johnsomor@gmail.com> a
écrit :

> Hi there Flint.
>
> Octavia fully supports using self-signed certificates and we use those
> in our gate tests.
> We do not allow non-TLS authenticated connections in the code, even
> for lab setups.
>
> This is a configuration issue or certificate file format issue. When
> the controller is attempting to access the controller local
> certificate file (likely the one we use to prove we are a valid
> controller to the amphora agent) it is finding a file without the
> required PEM format header. Check that your certificate files have the
> "-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
> format and just need to be converted).
>
> Also for reference, here are the minimal steps we use in our gate
> tests to setup the TLS certificates:
>
> https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305
>
> Michael
> On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com>
> wrote:
> >
> >
> > Hi guys,
> >
> > I continue to work on my Octavia integration using Kolla-Ansible and I'm
> facing a strange behavior.
> >
> > As for now I'm working on a POC using restricted HW and SW Capacities,
> I'm facing a strange issue when trying to launch a new load-balancer.
> >
> > When I create a new LB, would it be using CLI or WebUI, the amphora
> immediately disappear and the LB status switch to ERROR.
> >
> > When looking at logs and especially Worker logs, I see that the error
> seems to be related to the fact that the worker can't connect to the
> amphora because of a TLS Handshake issue which so trigger the contact
> timeout and rollback the amphora creation.
> >
> > Here is the worker.log relevant trace:
> >
> > 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-]
> Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
> > 2018-08-07 07:33:57.220 24 INFO
> octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB
> with id c20af002-1576-446e-b99f-7af607b8d885
> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-]
> Signing a certificate request using OpenSSL locally.
> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-]
> Using CA Certificate from config.
> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-]
> Using CA Private Key from config.
> > 2018-08-07 07:33:57.286 24 INFO octavia.certificates.generator.local [-]
> Using CA Private Key Passphrase from config.
> > 2018-08-07 07:34:04.074 24 INFO
> octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for
> amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id
> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer:
> bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> > 2018-08-07 07:34:04.253 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Port
> a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
> > 2018-08-07 07:34:19.656 24 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
> instance. Retrying.: ConnectTimeout:
> HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded
> with url: /0.5/plug/vip/192.168.56.100 (Caused by
> ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection
> object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect
> timeout=10.0)'))
> > 2018-08-07 07:34:24.673 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE'
> from state 'RUNNING'
> > 34 predecessors (most recent first):
> > Atom
> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>},
> 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885':
> <octavia.network.data_models.AmphoraNetworkConfig object at
> 0x7f4c284786d0>}}
> > |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE', 'state':
> 'SUCCESS', 'requires': {'loadbalancer_id':
> u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
> > |__Atom
> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]},
> 'provides': None}
> > |__Atom 'octavia.controller.worker.tasks.network_tasks.ApplyQos'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>],
> 'loadbalancer': <octavia.common.data_models.LoadBalancer object at
> 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
> > |__Atom
> 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention':
> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>},
> 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
> > |__Atom
> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip':
> <octavia.common.data_models.Vip object at 0x7f4c284956d0>,
> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
> > |__Atom
> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention':
> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>},
> 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
> > |__Flow 'octavia-new-loadbalancer-net-subflow'
> > |__Atom
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
> > |__Flow
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
> > |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides':
> None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>,
> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides':
> None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj':
> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885',
> 'build_type_priority': 40}, 'provides':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem'
> {'intention': 'EXECUTE', 'state': 'SUCCESS'}
> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides':
> u'c20af002-1576-446e-b99f-7af607b8d885'}
> > | |__Flow
> 'STANDALONE-octavia-create-amp-for-lb-subflow'
> > |
> |__Atom
> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> > |
> |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> > |
> |__Atom
> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> > |
> |__Flow 'octavia-create-loadbalancer-flow'
> > |__Atom
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb'
> {'intention': 'IGNORE', 'state': 'IGNORE'}
> > |__Atom
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora'
> {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
> > |__Flow
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
> > |__Atom
> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> > None}
> > |__Flow
> 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> > |__Atom
> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> > |__Flow
> 'octavia-create-loadbalancer-flow': Error: [.('PEM routines',
> 'PEM_read_bio', 'no start line'), ('SSL routines',
> 'SSL_CTX_use_certificate_file', 'PEM lib')]
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker Traceback (most recent call
> last):
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
> line 53, in _execute_task
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker result =
> task.execute(**arguments)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
> line 240, in execute
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker amphorae_network_config)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
> line 219, in execute
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker amphora, loadbalancer,
> amphorae_network_config)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 137, in post_vip_plug
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker net_info)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 388, in plug_vip
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker json=net_info)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 277, in request
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker r = _request(**reqargs)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker return self.request('POST',
> url, data=data, json=json, **kwargs)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in
> request
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker resp = self.send(prep,
> **send_kwargs)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker r = adapter.send(request,
> **kwargs)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker timeout=timeout
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 600, in urlopen
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker chunked=chunked)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 345, in _make_request
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker self._validate_conn(conn)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 844, in _validate_conn
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker conn.connect()
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py",
> line 326, in connect
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker ssl_context=context)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py",
> line 323, in ssl_wrap_socket
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> context.load_cert_chain(certfile, keyfile)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py",
> line 418, in load_cert_chain
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> self._ctx.use_certificate_file(certfile)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in
> use_certificate_file
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker _raise_current_error()
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in
> exception_from_error_queue
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker raise exception_type(errors)
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker Error: [.('PEM routines',
> 'PEM_read_bio', 'no start line'), ('SSL routines',
> 'SSL_CTX_use_certificate_file', 'PEM lib')]
> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> > 2018-08-07 07:34:24.684 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:24.687 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
> (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:24.691 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f)
> transitioned into state 'REVERTED' from state 'REVERTING'
> > 2018-08-07 07:34:24.694 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
> (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:24.716 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.ApplyQos'
> (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:24.719 24 WARNING
> octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for
> loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> > 2018-08-07 07:34:26.413 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.PlugVIP'
> (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:26.420 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
> (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:26.425 24 WARNING
> octavia.controller.worker.tasks.network_tasks [-] Deallocating vip
> 192.168.56.100
> > 2018-08-07 07:34:26.577 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security
> group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port
> a7bae53e-0bc6-4830-8c75-646a8baf2885
> > 2018-08-07 07:34:27.187 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security
> group 3d84ee39-1db9-475f-b048-9fe0f87201c1
> > 2018-08-07 07:34:27.803 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP'
> (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:27.807 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
> (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:27.810 24 WARNING
> octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role
> in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
> > 2018-08-07 07:34:27.816 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
> (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:27.819 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
> (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:27.821 24 WARNING
> octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora
> ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id
> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
> > 2018-08-07 07:34:27.826 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
> (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED'
> from state 'REVERTING'
> > 2018-08-07 07:34:27.828 24 WARNING
> octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora
> finalize.
> >
> > Is this a problem if I use self-signed CAcert ?
> > Is their a way to tell octavia to ignore SSL Error while working on a
> LAB environment?
> >
> > As usual, if you need further information feel free to ask.
> >
> > Thanks a lot guys.
> >
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
I’ll try to check the certificate format and make the appropriate change if
required or let you know if I’ve got something specific regarding that
topic.

Kind regards,
G.
Le mar. 14 août 2018 à 19:52, Flint WALRUS <gael.therond@gmail.com> a
écrit :

> Hi Michael, thanks a lot for your quick response once again!
> Le mar. 14 août 2018 à 18:21, Michael Johnson <johnsomor@gmail.com> a
> écrit :
>
>> Hi there Flint.
>>
>> Octavia fully supports using self-signed certificates and we use those
>> in our gate tests.
>> We do not allow non-TLS authenticated connections in the code, even
>> for lab setups.
>>
>> This is a configuration issue or certificate file format issue. When
>> the controller is attempting to access the controller local
>> certificate file (likely the one we use to prove we are a valid
>> controller to the amphora agent) it is finding a file without the
>> required PEM format header. Check that your certificate files have the
>> "-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
>> format and just need to be converted).
>>
>> Also for reference, here are the minimal steps we use in our gate
>> tests to setup the TLS certificates:
>>
>> https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305
>>
>> Michael
>> On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com>
>> wrote:
>> >
>> >
>> > Hi guys,
>> >
>> > I continue to work on my Octavia integration using Kolla-Ansible and
>> I'm facing a strange behavior.
>> >
>> > As for now I'm working on a POC using restricted HW and SW Capacities,
>> I'm facing a strange issue when trying to launch a new load-balancer.
>> >
>> > When I create a new LB, would it be using CLI or WebUI, the amphora
>> immediately disappear and the LB status switch to ERROR.
>> >
>> > When looking at logs and especially Worker logs, I see that the error
>> seems to be related to the fact that the worker can't connect to the
>> amphora because of a TLS Handshake issue which so trigger the contact
>> timeout and rollback the amphora creation.
>> >
>> > Here is the worker.log relevant trace:
>> >
>> > 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-]
>> Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
>> > 2018-08-07 07:33:57.220 24 INFO
>> octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB
>> with id c20af002-1576-446e-b99f-7af607b8d885
>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>> [-] Signing a certificate request using OpenSSL locally.
>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>> [-] Using CA Certificate from config.
>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>> [-] Using CA Private Key from config.
>> > 2018-08-07 07:33:57.286 24 INFO octavia.certificates.generator.local
>> [-] Using CA Private Key Passphrase from config.
>> > 2018-08-07 07:34:04.074 24 INFO
>> octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for
>> amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id
>> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer:
>> bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>> > 2018-08-07 07:34:04.253 24 INFO
>> octavia.network.drivers.neutron.allowed_address_pairs [-] Port
>> a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
>> > 2018-08-07 07:34:19.656 24 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
>> instance. Retrying.: ConnectTimeout:
>> HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded
>> with url: /0.5/plug/vip/192.168.56.100 (Caused by
>> ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection
>> object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect
>> timeout=10.0)'))
>> > 2018-08-07 07:34:24.673 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
>> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE'
>> from state 'RUNNING'
>> > 34 predecessors (most recent first):
>> > Atom
>> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>},
>> 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885':
>> <octavia.network.data_models.AmphoraNetworkConfig object at
>> 0x7f4c284786d0>}}
>> > |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE', 'state':
>> 'SUCCESS', 'requires': {'loadbalancer_id':
>> u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
>> > |__Atom
>> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
>> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]},
>> 'provides': None}
>> > |__Atom
>> 'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention':
>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
>> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>],
>> 'loadbalancer': <octavia.common.data_models.LoadBalancer object at
>> 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
>> > |__Atom
>> 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention':
>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>},
>> 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
>> > |__Atom
>> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip':
>> <octavia.common.data_models.Vip object at 0x7f4c284956d0>,
>> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
>> > |__Atom
>> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention':
>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>},
>> 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
>> > |__Flow 'octavia-new-loadbalancer-net-subflow'
>> > |__Atom
>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
>> > |__Flow
>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
>> > |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides':
>> None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>> u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
>> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>,
>> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides':
>> None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj':
>> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides':
>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
>> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
>> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
>> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885',
>> 'build_type_priority': 40}, 'provides':
>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
>> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
>> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS'}
>> > | |__Atom
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides':
>> u'c20af002-1576-446e-b99f-7af607b8d885'}
>> > |
>> |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
>> > |
>> |__Atom
>> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> None}
>> > |
>> |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>> > |
>> |__Atom
>> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> None}
>> > |
>> |__Flow 'octavia-create-loadbalancer-flow'
>> > |__Atom
>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb'
>> {'intention': 'IGNORE', 'state': 'IGNORE'}
>> > |__Atom
>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora'
>> {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
>> > |__Flow
>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
>> > |__Atom
>> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> > None}
>> > |__Flow
>> 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>> > |__Atom
>> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>> None}
>> > |__Flow
>> 'octavia-create-loadbalancer-flow': Error: [.('PEM routines',
>> 'PEM_read_bio', 'no start line'), ('SSL routines',
>> 'SSL_CTX_use_certificate_file', 'PEM lib')]
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker Traceback (most recent call
>> last):
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
>> line 53, in _execute_task
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker result =
>> task.execute(**arguments)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
>> line 240, in execute
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker amphorae_network_config)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
>> line 219, in execute
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker amphora, loadbalancer,
>> amphorae_network_config)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>> line 137, in post_vip_plug
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker net_info)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>> line 388, in plug_vip
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker json=net_info)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>> line 277, in request
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker r = _request(**reqargs)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker return self.request('POST',
>> url, data=data, json=json, **kwargs)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in
>> request
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker resp = self.send(prep,
>> **send_kwargs)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker r = adapter.send(request,
>> **kwargs)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker timeout=timeout
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>> line 600, in urlopen
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker chunked=chunked)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>> line 345, in _make_request
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker self._validate_conn(conn)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>> line 844, in _validate_conn
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker conn.connect()
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py",
>> line 326, in connect
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker ssl_context=context)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py",
>> line 323, in ssl_wrap_socket
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker
>> context.load_cert_chain(certfile, keyfile)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py",
>> line 418, in load_cert_chain
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker
>> self._ctx.use_certificate_file(certfile)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in
>> use_certificate_file
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker _raise_current_error()
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker File
>> "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in
>> exception_from_error_queue
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker raise exception_type(errors)
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker Error: [.('PEM routines',
>> 'PEM_read_bio', 'no start line'), ('SSL routines',
>> 'SSL_CTX_use_certificate_file', 'PEM lib')]
>> > 2018-08-07 07:34:24.673 24 ERROR
>> octavia.controller.worker.controller_worker
>> > 2018-08-07 07:34:24.684 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
>> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:24.687 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
>> (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:24.691 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f)
>> transitioned into state 'REVERTED' from state 'REVERTING'
>> > 2018-08-07 07:34:24.694 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
>> (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:24.716 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.network_tasks.ApplyQos'
>> (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:24.719 24 WARNING
>> octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for
>> loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>> > 2018-08-07 07:34:26.413 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.network_tasks.PlugVIP'
>> (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:26.420 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
>> (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:26.425 24 WARNING
>> octavia.controller.worker.tasks.network_tasks [-] Deallocating vip
>> 192.168.56.100
>> > 2018-08-07 07:34:26.577 24 INFO
>> octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security
>> group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port
>> a7bae53e-0bc6-4830-8c75-646a8baf2885
>> > 2018-08-07 07:34:27.187 24 INFO
>> octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security
>> group 3d84ee39-1db9-475f-b048-9fe0f87201c1
>> > 2018-08-07 07:34:27.803 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP'
>> (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:27.807 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
>> (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:27.810 24 WARNING
>> octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role
>> in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
>> > 2018-08-07 07:34:27.816 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
>> (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:27.819 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
>> (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:27.821 24 WARNING
>> octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora
>> ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id
>> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
>> > 2018-08-07 07:34:27.826 24 WARNING
>> octavia.controller.worker.controller_worker [-] Task
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
>> (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> > 2018-08-07 07:34:27.828 24 WARNING
>> octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora
>> finalize.
>> >
>> > Is this a problem if I use self-signed CAcert ?
>> > Is their a way to tell octavia to ignore SSL Error while working on a
>> LAB environment?
>> >
>> > As usual, if you need further information feel free to ask.
>> >
>> > Thanks a lot guys.
>> >
>> >
>> > _______________________________________________
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
Hi Michael,

Ok, it was indeed an issue with the create_certificate.sh script for centos
that indeed improperly created the client.pem certificate.

However now the amphora is responding with a 404 not found when the worker
is trying to post /v0.5/plug/vip/10.1.56.12

I know the amphora and the worker are correctly communicating as I can see
the amphora-proxy net namespace being set with the subnet ip as eth1 and
the vip as eth1:0

I did a tcpdump on each side (worker and amphora) and correctly see the
network two ways communication.

I checked the 9443 port and it is correctly binded to the gunicorn server
using the lb-mgmt-net ip of the amphora.

Is there any logs regarding the gunicorn server where I could check why
does the amphora is not able to found the api endpoint?
Le mar. 14 août 2018 à 19:53, Flint WALRUS <gael.therond@gmail.com> a
écrit :

> I’ll try to check the certificate format and make the appropriate change
> if required or let you know if I’ve got something specific regarding that
> topic.
>
> Kind regards,
> G.
> Le mar. 14 août 2018 à 19:52, Flint WALRUS <gael.therond@gmail.com> a
> écrit :
>
>> Hi Michael, thanks a lot for your quick response once again!
>> Le mar. 14 août 2018 à 18:21, Michael Johnson <johnsomor@gmail.com> a
>> écrit :
>>
>>> Hi there Flint.
>>>
>>> Octavia fully supports using self-signed certificates and we use those
>>> in our gate tests.
>>> We do not allow non-TLS authenticated connections in the code, even
>>> for lab setups.
>>>
>>> This is a configuration issue or certificate file format issue. When
>>> the controller is attempting to access the controller local
>>> certificate file (likely the one we use to prove we are a valid
>>> controller to the amphora agent) it is finding a file without the
>>> required PEM format header. Check that your certificate files have the
>>> "-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
>>> format and just need to be converted).
>>>
>>> Also for reference, here are the minimal steps we use in our gate
>>> tests to setup the TLS certificates:
>>>
>>> https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305
>>>
>>> Michael
>>> On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com>
>>> wrote:
>>> >
>>> >
>>> > Hi guys,
>>> >
>>> > I continue to work on my Octavia integration using Kolla-Ansible and
>>> I'm facing a strange behavior.
>>> >
>>> > As for now I'm working on a POC using restricted HW and SW Capacities,
>>> I'm facing a strange issue when trying to launch a new load-balancer.
>>> >
>>> > When I create a new LB, would it be using CLI or WebUI, the amphora
>>> immediately disappear and the LB status switch to ERROR.
>>> >
>>> > When looking at logs and especially Worker logs, I see that the error
>>> seems to be related to the fact that the worker can't connect to the
>>> amphora because of a TLS Handshake issue which so trigger the contact
>>> timeout and rollback the amphora creation.
>>> >
>>> > Here is the worker.log relevant trace:
>>> >
>>> > 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-]
>>> Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
>>> > 2018-08-07 07:33:57.220 24 INFO
>>> octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB
>>> with id c20af002-1576-446e-b99f-7af607b8d885
>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>>> [-] Signing a certificate request using OpenSSL locally.
>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>>> [-] Using CA Certificate from config.
>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local
>>> [-] Using CA Private Key from config.
>>> > 2018-08-07 07:33:57.286 24 INFO octavia.certificates.generator.local
>>> [-] Using CA Private Key Passphrase from config.
>>> > 2018-08-07 07:34:04.074 24 INFO
>>> octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for
>>> amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id
>>> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer:
>>> bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>>> > 2018-08-07 07:34:04.253 24 INFO
>>> octavia.network.drivers.neutron.allowed_address_pairs [-] Port
>>> a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
>>> > 2018-08-07 07:34:19.656 24 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
>>> instance. Retrying.: ConnectTimeout:
>>> HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded
>>> with url: /0.5/plug/vip/192.168.56.100 (Caused by
>>> ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection
>>> object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect
>>> timeout=10.0)'))
>>> > 2018-08-07 07:34:24.673 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
>>> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE'
>>> from state 'RUNNING'
>>> > 34 predecessors (most recent first):
>>> > Atom
>>> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>},
>>> 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885':
>>> <octavia.network.data_models.AmphoraNetworkConfig object at
>>> 0x7f4c284786d0>}}
>>> > |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE', 'state':
>>> 'SUCCESS', 'requires': {'loadbalancer_id':
>>> u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
>>> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]},
>>> 'provides': None}
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention':
>>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
>>> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>],
>>> 'loadbalancer': <octavia.common.data_models.LoadBalancer object at
>>> 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention':
>>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>},
>>> 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip':
>>> <octavia.common.data_models.Vip object at 0x7f4c284956d0>,
>>> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention':
>>> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>},
>>> 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
>>> > |__Flow 'octavia-new-loadbalancer-net-subflow'
>>> > |__Atom
>>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
>>> > |__Flow
>>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
>>> > |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>>> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides':
>>> None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>>> u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
>>> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>,
>>> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
>>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides':
>>> None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj':
>>> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides':
>>> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
>>> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
>>> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
>>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
>>> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
>>> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885',
>>> 'build_type_priority': 40}, 'provides':
>>> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
>>> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
>>> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS'}
>>> > | |__Atom
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides':
>>> u'c20af002-1576-446e-b99f-7af607b8d885'}
>>> > |
>>> |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
>>> > |
>>> |__Atom
>>> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> None}
>>> > |
>>> |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>>> > |
>>> |__Atom
>>> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> None}
>>> > |
>>> |__Flow 'octavia-create-loadbalancer-flow'
>>> > |__Atom
>>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb'
>>> {'intention': 'IGNORE', 'state': 'IGNORE'}
>>> > |__Atom
>>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora'
>>> {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
>>> > |__Flow
>>> 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
>>> > |__Atom
>>> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> > None}
>>> > |__Flow
>>> 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>>> > |__Atom
>>> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
>>> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
>>> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>> None}
>>> > |__Flow
>>> 'octavia-create-loadbalancer-flow': Error: [.('PEM routines',
>>> 'PEM_read_bio', 'no start line'), ('SSL routines',
>>> 'SSL_CTX_use_certificate_file', 'PEM lib')]
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker Traceback (most recent call
>>> last):
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
>>> line 53, in _execute_task
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker result =
>>> task.execute(**arguments)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
>>> line 240, in execute
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker amphorae_network_config)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
>>> line 219, in execute
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker amphora, loadbalancer,
>>> amphorae_network_config)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>>> line 137, in post_vip_plug
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker net_info)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>>> line 388, in plug_vip
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker json=net_info)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
>>> line 277, in request
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker r = _request(**reqargs)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker return self.request('POST',
>>> url, data=data, json=json, **kwargs)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in
>>> request
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker resp = self.send(prep,
>>> **send_kwargs)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker r = adapter.send(request,
>>> **kwargs)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker timeout=timeout
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>>> line 600, in urlopen
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker chunked=chunked)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>>> line 345, in _make_request
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker self._validate_conn(conn)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
>>> line 844, in _validate_conn
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker conn.connect()
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py",
>>> line 326, in connect
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker ssl_context=context)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py",
>>> line 323, in ssl_wrap_socket
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker
>>> context.load_cert_chain(certfile, keyfile)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py",
>>> line 418, in load_cert_chain
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker
>>> self._ctx.use_certificate_file(certfile)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in
>>> use_certificate_file
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker _raise_current_error()
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker File
>>> "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in
>>> exception_from_error_queue
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker raise exception_type(errors)
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker Error: [.('PEM routines',
>>> 'PEM_read_bio', 'no start line'), ('SSL routines',
>>> 'SSL_CTX_use_certificate_file', 'PEM lib')]
>>> > 2018-08-07 07:34:24.673 24 ERROR
>>> octavia.controller.worker.controller_worker
>>> > 2018-08-07 07:34:24.684 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
>>> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:24.687 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
>>> (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:24.691 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f)
>>> transitioned into state 'REVERTED' from state 'REVERTING'
>>> > 2018-08-07 07:34:24.694 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
>>> (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:24.716 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.network_tasks.ApplyQos'
>>> (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:24.719 24 WARNING
>>> octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for
>>> loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>>> > 2018-08-07 07:34:26.413 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.network_tasks.PlugVIP'
>>> (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:26.420 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
>>> (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:26.425 24 WARNING
>>> octavia.controller.worker.tasks.network_tasks [-] Deallocating vip
>>> 192.168.56.100
>>> > 2018-08-07 07:34:26.577 24 INFO
>>> octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security
>>> group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port
>>> a7bae53e-0bc6-4830-8c75-646a8baf2885
>>> > 2018-08-07 07:34:27.187 24 INFO
>>> octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security
>>> group 3d84ee39-1db9-475f-b048-9fe0f87201c1
>>> > 2018-08-07 07:34:27.803 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP'
>>> (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:27.807 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
>>> (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:27.810 24 WARNING
>>> octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role
>>> in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
>>> > 2018-08-07 07:34:27.816 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
>>> (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:27.819 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
>>> (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:27.821 24 WARNING
>>> octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora
>>> ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id
>>> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
>>> > 2018-08-07 07:34:27.826 24 WARNING
>>> octavia.controller.worker.controller_worker [-] Task
>>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
>>> (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED'
>>> from state 'REVERTING'
>>> > 2018-08-07 07:34:27.828 24 WARNING
>>> octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora
>>> finalize.
>>> >
>>> > Is this a problem if I use self-signed CAcert ?
>>> > Is their a way to tell octavia to ignore SSL Error while working on a
>>> LAB environment?
>>> >
>>> > As usual, if you need further information feel free to ask.
>>> >
>>> > Thanks a lot guys.
>>> >
>>> >
>>> > _______________________________________________
>>> > OpenStack-operators mailing list
>>> > OpenStack-operators@lists.openstack.org
>>> >
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
Yes, the amphora-agent logs to both the amphora-agent.log and syslog
in /var/log inside the amphora.

Michael
On Thu, Aug 16, 2018 at 1:43 PM Flint WALRUS <gael.therond@gmail.com> wrote:
>
> Hi Michael,
>
> Ok, it was indeed an issue with the create_certificate.sh script for centos that indeed improperly created the client.pem certificate.
>
> However now the amphora is responding with a 404 not found when the worker is trying to post /v0.5/plug/vip/10.1.56.12
>
> I know the amphora and the worker are correctly communicating as I can see the amphora-proxy net namespace being set with the subnet ip as eth1 and the vip as eth1:0
>
> I did a tcpdump on each side (worker and amphora) and correctly see the network two ways communication.
>
> I checked the 9443 port and it is correctly binded to the gunicorn server using the lb-mgmt-net ip of the amphora.
>
> Is there any logs regarding the gunicorn server where I could check why does the amphora is not able to found the api endpoint?
> Le mar. 14 août 2018 à 19:53, Flint WALRUS <gael.therond@gmail.com> a écrit :
>>
>> I’ll try to check the certificate format and make the appropriate change if required or let you know if I’ve got something specific regarding that topic.
>>
>> Kind regards,
>> G.
>> Le mar. 14 août 2018 à 19:52, Flint WALRUS <gael.therond@gmail.com> a écrit :
>>>
>>> Hi Michael, thanks a lot for your quick response once again!
>>> Le mar. 14 août 2018 à 18:21, Michael Johnson <johnsomor@gmail.com> a écrit :
>>>>
>>>> Hi there Flint.
>>>>
>>>> Octavia fully supports using self-signed certificates and we use those
>>>> in our gate tests.
>>>> We do not allow non-TLS authenticated connections in the code, even
>>>> for lab setups.
>>>>
>>>> This is a configuration issue or certificate file format issue. When
>>>> the controller is attempting to access the controller local
>>>> certificate file (likely the one we use to prove we are a valid
>>>> controller to the amphora agent) it is finding a file without the
>>>> required PEM format header. Check that your certificate files have the
>>>> "-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
>>>> format and just need to be converted).
>>>>
>>>> Also for reference, here are the minimal steps we use in our gate
>>>> tests to setup the TLS certificates:
>>>> https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305
>>>>
>>>> Michael
>>>> On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com> wrote:
>>>> >
>>>> >
>>>> > Hi guys,
>>>> >
>>>> > I continue to work on my Octavia integration using Kolla-Ansible and I'm facing a strange behavior.
>>>> >
>>>> > As for now I'm working on a POC using restricted HW and SW Capacities, I'm facing a strange issue when trying to launch a new load-balancer.
>>>> >
>>>> > When I create a new LB, would it be using CLI or WebUI, the amphora immediately disappear and the LB status switch to ERROR.
>>>> >
>>>> > When looking at logs and especially Worker logs, I see that the error seems to be related to the fact that the worker can't connect to the amphora because of a TLS Handshake issue which so trigger the contact timeout and rollback the amphora creation.
>>>> >
>>>> > Here is the worker.log relevant trace:
>>>> >
>>>> > 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint [-] Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
>>>> > 2018-08-07 07:33:57.220 24 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id c20af002-1576-446e-b99f-7af607b8d885
>>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
>>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
>>>> > 2018-08-07 07:33:57.285 24 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
>>>> > 2018-08-07 07:33:57.286 24 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
>>>> > 2018-08-07 07:34:04.074 24 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer: bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>>>> > 2018-08-07 07:34:04.253 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
>>>> > 2018-08-07 07:34:19.656 24 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.56.100 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect timeout=10.0)'))
>>>> > 2018-08-07 07:34:24.673 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug' (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE' from state 'RUNNING'
>>>> > 34 predecessors (most recent first):
>>>> > Atom 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}, 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885': <octavia.network.data_models.AmphoraNetworkConfig object at 0x7f4c284786d0>}}
>>>> > |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
>>>> > |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}, 'provides': None}
>>>> > |__Atom 'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
>>>> > |__Atom 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}, 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
>>>> > |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip': <octavia.common.data_models.Vip object at 0x7f4c284956d0>, 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
>>>> > |__Atom 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}, 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
>>>> > |__Flow 'octavia-new-loadbalancer-net-subflow'
>>>> > |__Atom 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
>>>> > |__Flow 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
>>>> > |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>, 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj': <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem': '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n', 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885', 'build_type_priority': 40}, 'provides': u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem': '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n', 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' {'intention': 'EXECUTE', 'state': 'SUCCESS'}
>>>> > | |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': u'c20af002-1576-446e-b99f-7af607b8d885'}
>>>> > | |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
>>>> > | |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
>>>> > | |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>>>> > | |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
>>>> > | |__Flow 'octavia-create-loadbalancer-flow'
>>>> > |__Atom 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb' {'intention': 'IGNORE', 'state': 'IGNORE'}
>>>> > |__Atom 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora' {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
>>>> > |__Flow 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
>>>> > |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
>>>> > None}
>>>> > |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
>>>> > |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides': None}
>>>> > |__Flow 'octavia-create-loadbalancer-flow': Error: [.('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 240, in execute
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker amphorae_network_config)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 219, in execute
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker amphora, loadbalancer, amphorae_network_config)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 137, in post_vip_plug
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker net_info)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 388, in plug_vip
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker json=net_info)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 277, in request
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker r = _request(**reqargs)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker return self.request('POST', url, data=data, json=json, **kwargs)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker resp = self.send(prep, **send_kwargs)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker r = adapter.send(request, **kwargs)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker timeout=timeout
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker chunked=chunked)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker self._validate_conn(conn)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker conn.connect()
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 326, in connect
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker ssl_context=context)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 323, in ssl_wrap_socket
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker context.load_cert_chain(certfile, keyfile)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 418, in load_cert_chain
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker self._ctx.use_certificate_file(certfile)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in use_certificate_file
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker _raise_current_error()
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker raise exception_type(errors)
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker Error: [.('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
>>>> > 2018-08-07 07:34:24.673 24 ERROR octavia.controller.worker.controller_worker
>>>> > 2018-08-07 07:34:24.684 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug' (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:24.687 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:24.691 24 WARNING octavia.controller.worker.controller_worker [-] Task 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:24.694 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData' (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:24.716 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.ApplyQos' (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:24.719 24 WARNING octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
>>>> > 2018-08-07 07:34:26.413 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.PlugVIP' (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:26.420 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:26.425 24 WARNING octavia.controller.worker.tasks.network_tasks [-] Deallocating vip 192.168.56.100
>>>> > 2018-08-07 07:34:26.577 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port a7bae53e-0bc6-4830-8c75-646a8baf2885
>>>> > 2018-08-07 07:34:27.187 24 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security group 3d84ee39-1db9-475f-b048-9fe0f87201c1
>>>> > 2018-08-07 07:34:27.803 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:27.807 24 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc' (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:27.810 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
>>>> > 2018-08-07 07:34:27.816 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb' (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:27.819 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora' (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:27.821 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
>>>> > 2018-08-07 07:34:27.826 24 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb' (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED' from state 'REVERTING'
>>>> > 2018-08-07 07:34:27.828 24 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora finalize.
>>>> >
>>>> > Is this a problem if I use self-signed CAcert ?
>>>> > Is their a way to tell octavia to ignore SSL Error while working on a LAB environment?
>>>> >
>>>> > As usual, if you need further information feel free to ask.
>>>> >
>>>> > Thanks a lot guys.
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > OpenStack-operators mailing list
>>>> > OpenStack-operators@lists.openstack.org
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [OCTAVIA][KOLLA] - Self signed CA/CERTS [ In reply to ]
Ok, I’ll have a look at the syslog logs as there was nothing but the 404
inside the agent logs.

I’ll not be able to get my hand on my lab until at least the middle of the
next week so don’t worry if I’m not coming back to you with my results.

It’s not that I solved it, just that I won’t get my lab available as I have
to share it to one of my colleagues working on keystone SSO signing :-)

Have a nice weekend and thanks again for your kind support.
Le sam. 18 août 2018 à 00:05, Michael Johnson <johnsomor@gmail.com> a
écrit :

> Yes, the amphora-agent logs to both the amphora-agent.log and syslog
> in /var/log inside the amphora.
>
> Michael
> On Thu, Aug 16, 2018 at 1:43 PM Flint WALRUS <gael.therond@gmail.com>
> wrote:
> >
> > Hi Michael,
> >
> > Ok, it was indeed an issue with the create_certificate.sh script for
> centos that indeed improperly created the client.pem certificate.
> >
> > However now the amphora is responding with a 404 not found when the
> worker is trying to post /v0.5/plug/vip/10.1.56.12
> >
> > I know the amphora and the worker are correctly communicating as I can
> see the amphora-proxy net namespace being set with the subnet ip as eth1
> and the vip as eth1:0
> >
> > I did a tcpdump on each side (worker and amphora) and correctly see the
> network two ways communication.
> >
> > I checked the 9443 port and it is correctly binded to the gunicorn
> server using the lb-mgmt-net ip of the amphora.
> >
> > Is there any logs regarding the gunicorn server where I could check why
> does the amphora is not able to found the api endpoint?
> > Le mar. 14 août 2018 à 19:53, Flint WALRUS <gael.therond@gmail.com> a
> écrit :
> >>
> >> I’ll try to check the certificate format and make the appropriate
> change if required or let you know if I’ve got something specific regarding
> that topic.
> >>
> >> Kind regards,
> >> G.
> >> Le mar. 14 août 2018 à 19:52, Flint WALRUS <gael.therond@gmail.com> a
> écrit :
> >>>
> >>> Hi Michael, thanks a lot for your quick response once again!
> >>> Le mar. 14 août 2018 à 18:21, Michael Johnson <johnsomor@gmail.com> a
> écrit :
> >>>>
> >>>> Hi there Flint.
> >>>>
> >>>> Octavia fully supports using self-signed certificates and we use those
> >>>> in our gate tests.
> >>>> We do not allow non-TLS authenticated connections in the code, even
> >>>> for lab setups.
> >>>>
> >>>> This is a configuration issue or certificate file format issue. When
> >>>> the controller is attempting to access the controller local
> >>>> certificate file (likely the one we use to prove we are a valid
> >>>> controller to the amphora agent) it is finding a file without the
> >>>> required PEM format header. Check that your certificate files have the
> >>>> "-----BEGIN CERTIFICATE-----" line (maybe they are in binary DER
> >>>> format and just need to be converted).
> >>>>
> >>>> Also for reference, here are the minimal steps we use in our gate
> >>>> tests to setup the TLS certificates:
> >>>>
> https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305
> >>>>
> >>>> Michael
> >>>> On Tue, Aug 14, 2018 at 4:54 AM Flint WALRUS <gael.therond@gmail.com>
> wrote:
> >>>> >
> >>>> >
> >>>> > Hi guys,
> >>>> >
> >>>> > I continue to work on my Octavia integration using Kolla-Ansible
> and I'm facing a strange behavior.
> >>>> >
> >>>> > As for now I'm working on a POC using restricted HW and SW
> Capacities, I'm facing a strange issue when trying to launch a new
> load-balancer.
> >>>> >
> >>>> > When I create a new LB, would it be using CLI or WebUI, the amphora
> immediately disappear and the LB status switch to ERROR.
> >>>> >
> >>>> > When looking at logs and especially Worker logs, I see that the
> error seems to be related to the fact that the worker can't connect to the
> amphora because of a TLS Handshake issue which so trigger the contact
> timeout and rollback the amphora creation.
> >>>> >
> >>>> > Here is the worker.log relevant trace:
> >>>> >
> >>>> > 2018-08-07 07:33:57.108 24 INFO octavia.controller.queue.endpoint
> [-] Creating load balancer 'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'...
> >>>> > 2018-08-07 07:33:57.220 24 INFO
> octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB
> with id c20af002-1576-446e-b99f-7af607b8d885
> >>>> > 2018-08-07 07:33:57.285 24 INFO
> octavia.certificates.generator.local [-] Signing a certificate request
> using OpenSSL locally.
> >>>> > 2018-08-07 07:33:57.285 24 INFO
> octavia.certificates.generator.local [-] Using CA Certificate from config.
> >>>> > 2018-08-07 07:33:57.285 24 INFO
> octavia.certificates.generator.local [-] Using CA Private Key from config.
> >>>> > 2018-08-07 07:33:57.286 24 INFO
> octavia.certificates.generator.local [-] Using CA Private Key Passphrase
> from config.
> >>>> > 2018-08-07 07:34:04.074 24 INFO
> octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for
> amphora: c20af002-1576-446e-b99f-7af607b8d885 with compute id
> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1 for load balancer:
> bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> >>>> > 2018-08-07 07:34:04.253 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Port
> a7bae53e-0bc6-4830-8c75-646a8baf2885 already exists. Nothing to be done.
> >>>> > 2018-08-07 07:34:19.656 24 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
> instance. Retrying.: ConnectTimeout:
> HTTPSConnectionPool(host='10.1.56.103', port=9443): Max retries exceeded
> with url: /0.5/plug/vip/192.168.56.100 (Caused by
> ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection
> object at 0x7f4c28415c50>, 'Connection to 10.1.56.103 timed out. (connect
> timeout=10.0)'))
> >>>> > 2018-08-07 07:34:24.673 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'FAILURE'
> from state 'RUNNING'
> >>>> > 34 predecessors (most recent first):
> >>>> > Atom
> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>},
> 'provides': {u'c20af002-1576-446e-b99f-7af607b8d885':
> <octavia.network.data_models.AmphoraNetworkConfig object at
> 0x7f4c284786d0>}}
> >>>> > |__Atom 'reload-lb-after-plug-vip' {'intention': 'EXECUTE',
> 'state': 'SUCCESS', 'requires': {'loadbalancer_id':
> u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478d90>}
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]},
> 'provides': None}
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.network_tasks.ApplyQos' {'intention':
> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amps_data':
> [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>],
> 'loadbalancer': <octavia.common.data_models.LoadBalancer object at
> 0x7f4c2845fe10>, 'update_dict': {'topology': 'SINGLE'}}, 'provides': None}
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.network_tasks.PlugVIP' {'intention':
> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>},
> 'provides': [<octavia.common.data_models.Amphora object at 0x7f4c285165d0>]}
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'vip':
> <octavia.common.data_models.Vip object at 0x7f4c284956d0>,
> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c2845fe10>}
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention':
> 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>},
> 'provides': <octavia.common.data_models.Vip object at 0x7f4c284956d0>}
> >>>> > |__Flow 'octavia-new-loadbalancer-net-subflow'
> >>>> > |__Atom
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> <octavia.common.data_models.LoadBalancer object at 0x7f4c28478110>}
> >>>> > |__Flow
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow'
> >>>> > |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}, 'provides':
> None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c28478190>}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>,
> 'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amphora-finalize'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}, 'provides':
> None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_obj':
> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c28478a50>}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides':
> <octavia.common.data_models.Amphora object at 0x7f4c2845fa10>}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id':
> u'c20af002-1576-446e-b99f-7af607b8d885', 'compute_id':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}, 'provides': None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885',
> 'build_type_priority': 40}, 'provides':
> u'3bbabfa6-366f-46a4-8fb2-1ec7158e19f1'}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem':
> '-----BEGIN CERTIFICATE-----\n REDACTED \n-----END RSA PRIVATE KEY-----\n',
> 'amphora_id': u'c20af002-1576-446e-b99f-7af607b8d885'}, 'provides': None}
> >>>> > | |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem'
> {'intention': 'EXECUTE', 'state': 'SUCCESS'}
> >>>> > |
> |__Atom
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides':
> u'c20af002-1576-446e-b99f-7af607b8d885'}
> >>>> > |
> |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
> >>>> > |
> |__Atom
> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> >>>> > |
> |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> >>>> > |
> |__Atom
> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> >>>> > |
> |__Flow 'octavia-create-loadbalancer-flow'
> >>>> > |__Atom
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-mark-amp-standalone-indb'
> {'intention': 'IGNORE', 'state': 'IGNORE'}
> >>>> > |__Atom
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow-octavia-reload-amphora'
> {'intention': 'IGNORE', 'state': 'IGNORE', 'requires': {'amphora_id': None}}
> >>>> > |__Flow
> 'STANDALONE-octavia-post-map-amp-to-lb-subflow'
> >>>> > |__Atom
> 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> >>>> > None}
> >>>> > |__Flow
> 'STANDALONE-octavia-get-amphora-for-lb-subflow'
> >>>> > |__Atom
> 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask'
> {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires':
> {'loadbalancer_id': u'bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e'}, 'provides':
> None}
> >>>> > |__Flow
> 'octavia-create-loadbalancer-flow': Error: [.('PEM routines',
> 'PEM_read_bio', 'no start line'), ('SSL routines',
> 'SSL_CTX_use_certificate_file', 'PEM lib')]
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker Traceback (most recent call
> last):
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
> line 53, in _execute_task
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker result =
> task.execute(**arguments)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
> line 240, in execute
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker amphorae_network_config)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py",
> line 219, in execute
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker amphora, loadbalancer,
> amphorae_network_config)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 137, in post_vip_plug
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker net_info)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 388, in plug_vip
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker json=net_info)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py",
> line 277, in request
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker r = _request(**reqargs)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 565, in post
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker return self.request('POST',
> url, data=data, json=json, **kwargs)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in
> request
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker resp = self.send(prep,
> **send_kwargs)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker r = adapter.send(request,
> **kwargs)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/adapters.py", line 438, in send
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker timeout=timeout
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 600, in urlopen
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker chunked=chunked)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 345, in _make_request
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker self._validate_conn(conn)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 844, in _validate_conn
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker conn.connect()
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py",
> line 326, in connect
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker ssl_context=context)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py",
> line 323, in ssl_wrap_socket
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> context.load_cert_chain(certfile, keyfile)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py",
> line 418, in load_cert_chain
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> self._ctx.use_certificate_file(certfile)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 817, in
> use_certificate_file
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker _raise_current_error()
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker File
> "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 54, in
> exception_from_error_queue
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker raise exception_type(errors)
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker Error: [.('PEM routines',
> 'PEM_read_bio', 'no start line'), ('SSL routines',
> 'SSL_CTX_use_certificate_file', 'PEM lib')]
> >>>> > 2018-08-07 07:34:24.673 24 ERROR
> octavia.controller.worker.controller_worker
> >>>> > 2018-08-07 07:34:24.684 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug'
> (c86bbab6-87d5-4930-8832-5511d42efe3e) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:24.687 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs'
> (1e329fa2-b7c3-4fe2-93f0-d565a18cdbba) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:24.691 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'reload-lb-after-plug-vip' (842fb766-dd6f-4b3c-936a-7a5baa82c64f)
> transitioned into state 'REVERTED' from state 'REVERTING'
> >>>> > 2018-08-07 07:34:24.694 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.database_tasks.UpdateAmphoraVIPData'
> (761da17b-4655-46a9-9d67-cb7816c7ea0c) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:24.716 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.ApplyQos'
> (fb40f555-1f0a-48fc-b377-f9e791077f65) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:24.719 24 WARNING
> octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP for
> loadbalancer id bf7ab6e4-081a-4b4d-b7a0-c176a9cb995e
> >>>> > 2018-08-07 07:34:26.413 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.PlugVIP'
> (ae486972-6e98-4036-9e20-85f335058074) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:26.420 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation'
> (79391dee-6011-4145-b544-499e0a632ca1) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:26.425 24 WARNING
> octavia.controller.worker.tasks.network_tasks [-] Deallocating vip
> 192.168.56.100
> >>>> > 2018-08-07 07:34:26.577 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security
> group 3d84ee39-1db9-475f-b048-9fe0f87201c1 from port
> a7bae53e-0bc6-4830-8c75-646a8baf2885
> >>>> > 2018-08-07 07:34:27.187 24 INFO
> octavia.network.drivers.neutron.allowed_address_pairs [-] Deleted security
> group 3d84ee39-1db9-475f-b048-9fe0f87201c1
> >>>> > 2018-08-07 07:34:27.803 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia.controller.worker.tasks.network_tasks.AllocateVIP'
> (7edf30ee-4338-4725-a86e-e45c0aa0aa58) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:27.807 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'octavia-post-loadbalancer-amp_association-subflow-octavia-post-loadbalancer-amp_association-subflow-reload-lb-after-amp-assoc'
> (64ac1f84-f8ec-4cc1-b3c8-f18ac8474d73) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:27.810 24 WARNING
> octavia.controller.worker.tasks.database_tasks [-] Reverting amphora role
> in DB for amp id c20af002-1576-446e-b99f-7af607b8d885
> >>>> > 2018-08-07 07:34:27.816 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amp-standalone-indb'
> (2db823a7-c4ac-4622-824b-b709c96b554a) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:27.819 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-reload-amphora'
> (86219bac-efd2-4d1f-8141-818f1a5bc6f5) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:27.821 24 WARNING
> octavia.controller.worker.tasks.database_tasks [-] Reverting mark amphora
> ready in DB for amp id c20af002-1576-446e-b99f-7af607b8d885 and compute id
> 3bbabfa6-366f-46a4-8fb2-1ec7158e19f1
> >>>> > 2018-08-07 07:34:27.826 24 WARNING
> octavia.controller.worker.controller_worker [-] Task
> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-allocated-indb'
> (baf58e71-eef6-41e0-9bf3-ab9f9554ace2) transitioned into state 'REVERTED'
> from state 'REVERTING'
> >>>> > 2018-08-07 07:34:27.828 24 WARNING
> octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting amphora
> finalize.
> >>>> >
> >>>> > Is this a problem if I use self-signed CAcert ?
> >>>> > Is their a way to tell octavia to ignore SSL Error while working on
> a LAB environment?
> >>>> >
> >>>> > As usual, if you need further information feel free to ask.
> >>>> >
> >>>> > Thanks a lot guys.
> >>>> >
> >>>> >
> >>>> > _______________________________________________
> >>>> > OpenStack-operators mailing list
> >>>> > OpenStack-operators@lists.openstack.org
> >>>> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>