Mailing List Archive

Re: Fwd: Nova hypervisor uuid
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
I am sure kvm nodes names are note changed.
Tables where uuid are duplicated are:
dataresource_providers in nova_api db
compute_nodes in nova db
Regards
Ignazio

Il 28/Nov/2018 11:09 AM, "Gianpiero Ardissono" <gp.ardissono@gmail.com> ha
scritto:

>
> ---------- Forwarded message ---------
> From: Matt Riedemann <mriedemos@gmail.com>
> Date: mar 27 nov 2018, 19:03
> Subject: Re: [Openstack-operators] Nova hypervisor uuid
> To: <openstack-operators@lists.openstack.org>
>
>
> On 11/27/2018 11:32 AM, Ignazio Cassano wrote:
> > Hi All,
> > Please anyone know where hypervisor uuid is retrived?
> > Sometime updating kmv nodes with yum update it changes and in nova
> > database 2 uuids are assigned to the same node.
> > regards
> > Ignazio
> >
> >
> >
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
> To be clear, do you mean the computes_nodes.uuid column value in the
> cell database? Which is also used for the GET /os-hypervisors response
> 'id' value if using microversion >= 2.53. If so, that is generated
> randomly* when the compute_nodes table record is created:
>
>
> https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L588
>
>
> https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/objects/compute_node.py#L312
>
> When you hit this problem, are you sure the hostname on the compute host
> is not changing? Because when nova-compute starts up, it should look for
> the existing compute node record by host name and node name, which for
> the libvirt driver should be the same. That lookup code is here:
>
>
> https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L815
>
> So the only way nova-compute should create a new compute_nodes table
> record for the same host is if the host/node name changes during the
> upgrade. Is the deleted value in the database the same (0) for both of
> those records?
>
> * The exception to this is for the ironic driver which re-uses the
> ironic node uuid as of this change:
> https://review.openstack.org/#/c/571535/
>
> --
>
> Thanks,
>
> Matt
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
Re: Fwd: Nova hypervisor uuid [ In reply to ]
On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
> Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
> I am sure kvm nodes names are note changed.
> Tables where uuid are duplicated are:
> dataresource_providers in nova_api db
> compute_nodes in nova db
> Regards
> Ignazio

It would be easier if you simply dumped the result of a select query on
the compute_nodes table where the duplicate nodes exist (you said
duplicate UUIDs but I think you mean duplicate host/node names with
different UUIDs, correct?).

There is a unique constraint on host/hypervisor_hostname (nodename)/deleted:

schema.UniqueConstraint(
'host', 'hypervisor_hostname', 'deleted',
name="uniq_compute_nodes0host0hypervisor_hostname0deleted"),

So I'm wondering if the deleted field is not 0 on one of those because
if one is marked as deleted, then the compute service will create a new
compute_nodes table record on startup (and associated resource provider).

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: Fwd: Nova hypervisor uuid [ In reply to ]
Hello Mattm
Yes I mean sometimes I have same host/node names with different uuid in
compute_nodes table in nova database
I must delete nodes with uuid those not match with nova-hypervisor list
command.
At this time I have the following:
MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes;
+---------------------+--------------------------------------+---------+
| hypervisor_hostname | uuid | deleted |
+---------------------+--------------------------------------+---------+
| tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 |
| tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 |
+---------------------+--------------------------------------+---------+
2 rows in set (0,00 sec)


But sometimes old uuid are inserted in the table .
I deleted again them.
I restarted kvm nodes and now the table is ok.
I also restarded each controller and the tables is ok.
I do not know because 3 days ago I had same compute nodes names with
different uuids.

Thanks and Regards
Ignazio

Il giorno mer 28 nov 2018 alle ore 17:54 Matt Riedemann <mriedemos@gmail.com>
ha scritto:

> On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
> > Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
> > I am sure kvm nodes names are note changed.
> > Tables where uuid are duplicated are:
> > dataresource_providers in nova_api db
> > compute_nodes in nova db
> > Regards
> > Ignazio
>
> It would be easier if you simply dumped the result of a select query on
> the compute_nodes table where the duplicate nodes exist (you said
> duplicate UUIDs but I think you mean duplicate host/node names with
> different UUIDs, correct?).
>
> There is a unique constraint on host/hypervisor_hostname
> (nodename)/deleted:
>
> schema.UniqueConstraint(
> 'host', 'hypervisor_hostname', 'deleted',
> name="uniq_compute_nodes0host0hypervisor_hostname0deleted"),
>
> So I'm wondering if the deleted field is not 0 on one of those because
> if one is marked as deleted, then the compute service will create a new
> compute_nodes table record on startup (and associated resource provider).
>
> --
>
> Thanks,
>
> Matt
>
Re: Fwd: Nova hypervisor uuid [ In reply to ]
On 11/29/2018 12:49 AM, Ignazio Cassano wrote:
> Hello Mattm
> Yes I mean sometimes I have same host/node names with different uuid in
> compute_nodes table in nova database
> I must delete nodes with uuid those not match with nova-hypervisor list
> command.
> At this time I have the following:
> MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes;
> +---------------------+--------------------------------------+---------+
> | hypervisor_hostname | uuid                                 | deleted |
> +---------------------+--------------------------------------+---------+
> | tst2-kvm02          | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d |       0 |
> | tst2-kvm01          | ce27803b-06cd-44a7-b927-1fa42c813b0f |       0 |
> +---------------------+--------------------------------------+---------+
> 2 rows in set (0,00 sec)
>
>
> But sometimes old uuid are inserted in the table .
> I deleted again them.
> I restarted kvm nodes and now the table is ok.
> I also restarded each controller and the tables is ok.
> I do not know because 3 days ago I had same compute nodes names with
> different uuids.
>
> Thanks and Regards
> Ignazio

OK I guess if it happens again, please get the
host/hypervisor_hostname/uuid/deleted values from the compute_nodes
table before you cleanup any entries.

Also, when you're deleting the resources from the DB, are you doing it
in the DB directly or via the DELETE /os-services/{service_id} API?
Because the latter cleans up other related resources to the nova-compute
service (the services table record, the compute_nodes table record, the
related resource_providers table record in placement, and the
host_mappings table record in the nova API DB). The resource
provider/host mappings cleanup when deleting a compute service is a more
recent bug fix though which depending on your release you might not have:

https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: Fwd: Nova hypervisor uuid [ In reply to ]
Hi Matt,
I did in the DB directly.
I am using queens now.
Any python client command to delete hold records or I must use api ?

Thanks & Regards
Ignazio


Il giorno gio 29 nov 2018 alle ore 16:28 Matt Riedemann <mriedemos@gmail.com>
ha scritto:

> On 11/29/2018 12:49 AM, Ignazio Cassano wrote:
> > Hello Mattm
> > Yes I mean sometimes I have same host/node names with different uuid in
> > compute_nodes table in nova database
> > I must delete nodes with uuid those not match with nova-hypervisor list
> > command.
> > At this time I have the following:
> > MariaDB [nova]> select hypervisor_hostname,uuid,deleted from
> compute_nodes;
> > +---------------------+--------------------------------------+---------+
> > | hypervisor_hostname | uuid | deleted |
> > +---------------------+--------------------------------------+---------+
> > | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 |
> > | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 |
> > +---------------------+--------------------------------------+---------+
> > 2 rows in set (0,00 sec)
> >
> >
> > But sometimes old uuid are inserted in the table .
> > I deleted again them.
> > I restarted kvm nodes and now the table is ok.
> > I also restarded each controller and the tables is ok.
> > I do not know because 3 days ago I had same compute nodes names with
> > different uuids.
> >
> > Thanks and Regards
> > Ignazio
>
> OK I guess if it happens again, please get the
> host/hypervisor_hostname/uuid/deleted values from the compute_nodes
> table before you cleanup any entries.
>
> Also, when you're deleting the resources from the DB, are you doing it
> in the DB directly or via the DELETE /os-services/{service_id} API?
> Because the latter cleans up other related resources to the nova-compute
> service (the services table record, the compute_nodes table record, the
> related resource_providers table record in placement, and the
> host_mappings table record in the nova API DB). The resource
> provider/host mappings cleanup when deleting a compute service is a more
> recent bug fix though which depending on your release you might not have:
>
> https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80
>
> --
>
> Thanks,
>
> Matt
>
Re: Fwd: Nova hypervisor uuid [ In reply to ]
On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
> I did in the DB directly.
> I am using queens now.
> Any python client command to delete hold records or I must use api ?

You can use the CLI:

https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: Fwd: Nova hypervisor uuid [ In reply to ]
Many thks Matt.
If the issue will happen again
I hope openstack commands will show duplicate entries and let me clean them.
When happened nova-hypervisor list command did not show duplcated entries
but I saw them only in the database.
Regards
Ignazio

Il giorno Gio 29 Nov 2018 17:28 Matt Riedemann <mriedemos@gmail.com> ha
scritto:

> On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
> > I did in the DB directly.
> > I am using queens now.
> > Any python client command to delete hold records or I must use api ?
>
> You can use the CLI:
>
>
> https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete
>
>
> https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete
>
> --
>
> Thanks,
>
> Matt
>