Mailing List Archive: [Octavia] SSL errors polling amphorae and missing tenant network interface

[Octavia] SSL errors polling amphorae and missing tenant network interface

Oct 19, 2018, 11:39 AM

Post #1 of 3 (650 views)

I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.

Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and eventually fails. The error on the controller
side is:

2018-10-19 14:17:39.181 26 ERROR
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
retries (currently set to 300) exhausted. The amphora is unavailable.
Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by
SSLError(SSLError("bad handshake: Error([.('rsa routines',
'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
'tls_process_server_certificate', 'certificate verify
failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15
(Caused by SSLError(SSLError("bad handshake: Error([.('rsa routines',
'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
'tls_process_server_certificate', 'certificate verify
failed')],)",),))

On the amphora side I see:
[2018-10-19 17:52:54 +0000] [1331] [DEBUG] Error processing SSL request.
[2018-10-19 17:52:54 +0000] [1331] [DEBUG] Invalid request from
ip=::ffff:10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake
failure (_ssl.c:1754)

I've generated certificates both with the script in the Octavia git
repo, and with the Openstack Ansible playbook. I can see that they are
present in /etc/octavia/certs.

I'm using the Kolla (Queens) containers for the control plane so I'm
sure I've satisfied all the python library constraints.

Issue 2:
I"m not sure how it gets configured, but the tenant network interface
(ens6) never comes up. I can spawn other instances on that network
with no issue, and I can see that Neutron has the port attached to the
instance. However, in the instance this is all I get:

ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast
state UP group default qlen 1000
link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe30:c460/64 scope link
valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff

There's no evidence of the interface anywhere else including udev rules.

Any help with either or both issues would be greatly appreciated.

Cheers,
Erik

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Octavia] SSL errors polling amphorae and missing tenant network interface [ In reply to ]

gael.therond at gmail

Oct 19, 2018, 4:47 PM

Post #2 of 3 (649 views)

Permalink

Hi eric!

Glad I’m not the only one having this issue with the ssl communication
between the amphora and the CP.

Even if I don’t yet get a clear answer regarding that issue, I think your
second issue is not an issue as the interface is mounted on a namespace and
so you’ll need to list all nic even those from namespace.

Use an ip netns ls to get the namespace.

Hope it will help.

Le ven. 19 oct. 2018 à 20:40, Erik McCormick <emccormick@cirrusseven.com> a
écrit :

> I've been wrestling with getting Octavia up and running and have
> become stuck on two issues. I'm hoping someone has run into these
> before. My google foo has come up empty.
>
> Issue 1:
> When the Octavia controller tries to poll the amphora instance, it
> tries repeatedly and eventually fails. The error on the controller
> side is:
>
> 2018-10-19 14:17:39.181 26 ERROR
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
> retries (currently set to 300) exhausted. The amphora is unavailable.
> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
> exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by
> SSLError(SSLError("bad handshake: Error([.('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
> port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15
> (Caused by SSLError(SSLError("bad handshake: Error([.('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),))
>
> On the amphora side I see:
> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Error processing SSL request.
> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Invalid request from
> ip=::ffff:10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake
> failure (_ssl.c:1754)
>
> I've generated certificates both with the script in the Octavia git
> repo, and with the Openstack Ansible playbook. I can see that they are
> present in /etc/octavia/certs.
>
> I'm using the Kolla (Queens) containers for the control plane so I'm
> sure I've satisfied all the python library constraints.
>
> Issue 2:
> I"m not sure how it gets configured, but the tenant network interface
> (ens6) never comes up. I can spawn other instances on that network
> with no issue, and I can see that Neutron has the port attached to the
> instance. However, in the instance this is all I get:
>
> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast
> state UP group default qlen 1000
> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
> inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3
> valid_lft forever preferred_lft forever
> inet6 fe80::f816:3eff:fe30:c460/64 scope link
> valid_lft forever preferred_lft forever
> 3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
> default qlen 1000
> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff
>
> There's no evidence of the interface anywhere else including udev rules.
>
> Any help with either or both issues would be greatly appreciated.
>
> Cheers,
> Erik
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

Re: [Octavia] SSL errors polling amphorae and missing tenant network interface [ In reply to ]

tobias.urdin at binero

Oct 22, 2018, 1:28 AM

Post #3 of 3 (647 views)

Permalink

Hello,
Seems we are quite a few having difficulties getting it to work.

I missed adding operators ML to my previous reply, sent it again.

I'm at the point where SSL pretty much becomes a hassle for operations,
if there was an option to just
go with a shared secret I would've done a while ago, which probably says
a lot about the amount of time on this.

Best regards
Tobias

On 10/20/2018 01:58 AM, Gaël THEROND wrote:
> Hi eric!
>
> Glad I’m not the only one having this issue with the ssl communication
> between the amphora and the CP.
>
> Even if I don’t yet get a clear answer regarding that issue, I think
> your second issue is not an issue as the interface is mounted on a
> namespace and so you’ll need to list all nic even those from namespace.
>
> Use an ip netns ls to get the namespace.
>
> Hope it will help.
>
> Le ven. 19 oct. 2018 à 20:40, Erik McCormick
> <emccormick@cirrusseven.com <mailto:emccormick@cirrusseven.com>> a écrit :
>
> I've been wrestling with getting Octavia up and running and have
> become stuck on two issues. I'm hoping someone has run into these
> before. My google foo has come up empty.
>
> Issue 1:
> When the Octavia controller tries to poll the amphora instance, it
> tries repeatedly and eventually fails. The error on the controller
> side is:
>
> 2018-10-19 14:17:39.181 26 ERROR
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
> retries (currently set to 300) exhausted. The amphora is unavailable.
> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
> exceeded with url: /0.5/plug/vip/10.250.20.15
> <http://10.250.20.15> (Caused by
> SSLError(SSLError("bad handshake: Error([.('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
> port=9443): Max retries exceeded with url:
> /0.5/plug/vip/10.250.20.15 <http://10.250.20.15>
> (Caused by SSLError(SSLError("bad handshake: Error([.('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),))
>
> On the amphora side I see:
> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Error processing SSL
> request.
> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Invalid request from
> ip=::ffff:10.7.0.40 <http://10.7.0.40>: [SSL:
> SSL_HANDSHAKE_FAILURE] ssl handshake
> failure (_ssl.c:1754)
>
> I've generated certificates both with the script in the Octavia git
> repo, and with the Openstack Ansible playbook. I can see that they are
> present in /etc/octavia/certs.
>
> I'm using the Kolla (Queens) containers for the control plane so I'm
> sure I've satisfied all the python library constraints.
>
> Issue 2:
> I"m not sure how it gets configured, but the tenant network interface
> (ens6) never comes up. I can spawn other instances on that network
> with no issue, and I can see that Neutron has the port attached to the
> instance. However, in the instance this is all I get:
>
> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast
> state UP group default qlen 1000
> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
> inet 10.7.0.112/16 <http://10.7.0.112/16> brd 10.7.255.255
> scope global ens3
> valid_lft forever preferred_lft forever
> inet6 fe80::f816:3eff:fe30:c460/64 scope link
> valid_lft forever preferred_lft forever
> 3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
> default qlen 1000
> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff
>
> There's no evidence of the interface anywhere else including udev
> rules.
>
> Any help with either or both issues would be greatly appreciated.
>
> Cheers,
> Erik
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> <mailto:OpenStack-operators@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>