Mailing List Archive

On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann <mriedemos@gmail.com> wrote:
>
> On 9/5/2018 8:47 AM, Mohammed Naser wrote:
> > Could placement not do what happened for a while when the nova_api
> > database was created?
>
> Can you be more specific? I'm having a brain fart here and not
> remembering what you are referring to with respect to the nova_api DB.

I think there was a period in time where the nova_api database was created
where entires would try to get pulled out from the original nova database and
then checking nova_api if it doesn't exist afterwards (or vice versa). One
of the cases that this was done to deal with was for things like instance types
or flavours.

I don't know the exact details but I know that older instance types exist in
the nova db and the newer ones are sitting in nova_api. Something along
those lines?

> >
> > I say this because I know that moving the database is a huge task for
> > us, considering how big it can be in certain cases for us, and it
> > means control plane outage too
>
> I'm pretty sure you were in the room in YVR when we talked about how
> operators were going to do the database migration and were mostly OK
> with what was discussed, which was a lot will just copy and take the
> downtime (I think CERN said around 10 minutes for them, but they aren't
> a public cloud either), but others might do something more sophisticated
> and nova shouldn't try to pick the best fit for all.

If we're provided the list of tables used by placement, we could considerably
make the downtime smaller because we don't have to pull in the other huge
tables like instances/build requests/etc

What happens if things like server deletes happen while the placement service
is down?

> I'm definitely interested in what you do plan to do for the database
> migration to minimize downtime.

At this point, I'm thinking turn off placement, setup the new one, do
the migration
of the placement-specific tables (this can be a straightforward documented task
OR it would be awesome if it was a placement command (something along
the lines of `placement-manage db import_from_nova`) which would import all
the right things

The idea of having a command would be *extremely* useful for deployment tools
in automating the process and it also allows the placement team to selectively
decide what they want to onboard?

Just throwing ideas here.

> +openstack-operators ML since this is an operators discussion now.
>
> --
>
> Thanks,
>
> Matt



--
Mohammed Naser — vexxhost
-----------------------------------------------------
D. 514-316-8872
D. 800-910-1726 ext. 200
E. mnaser@vexxhost.com
W. http://vexxhost.com

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 9/5/2018 10:03 AM, Mohammed Naser wrote:
> On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann<mriedemos@gmail.com> wrote:
>> On 9/5/2018 8:47 AM, Mohammed Naser wrote:
>>> Could placement not do what happened for a while when the nova_api
>>> database was created?
>> Can you be more specific? I'm having a brain fart here and not
>> remembering what you are referring to with respect to the nova_api DB.
> I think there was a period in time where the nova_api database was created
> where entires would try to get pulled out from the original nova database and
> then checking nova_api if it doesn't exist afterwards (or vice versa). One
> of the cases that this was done to deal with was for things like instance types
> or flavours.
>
> I don't know the exact details but I know that older instance types exist in
> the nova db and the newer ones are sitting in nova_api. Something along
> those lines?

Yeah that more about supporting online data migrations *within* nova
where new records were created in the API DB and old records would be
looked up in both the API DB and then if not found there, in the cell
(traditional nova DB). But you'd also be running the "nova-manage db
online_data_migrations" CLI to force the migration of the records from
the cell DB to the API DB.

With Placement split out of nova, we can't really do that. You could
point placement at the nova_api DB so it can pull existing records, but
it would continue to create new records in the nova_api DB rather than
the placement DB and at some point you have to make that data migration.
Maybe you were thinking something like have temporary fallback code in
placement such that if a record isn't found in the placement database,
it queries a configured nova_api database? That'd be a ton of work at
this point, and if it was something we were going to do, we should have
agreed on that in YVR several months ago, definitely pre-extraction.

>
>>> I say this because I know that moving the database is a huge task for
>>> us, considering how big it can be in certain cases for us, and it
>>> means control plane outage too
>> I'm pretty sure you were in the room in YVR when we talked about how
>> operators were going to do the database migration and were mostly OK
>> with what was discussed, which was a lot will just copy and take the
>> downtime (I think CERN said around 10 minutes for them, but they aren't
>> a public cloud either), but others might do something more sophisticated
>> and nova shouldn't try to pick the best fit for all.
> If we're provided the list of tables used by placement, we could considerably
> make the downtime smaller because we don't have to pull in the other huge
> tables like instances/build requests/etc

There are no instances records in the API DB, maybe you mean
instance_mappings? But yes I get the point.

>
> What happens if things like server deletes happen while the placement service
> is down?

The DELETE /allocations/{consumer_id} requests from nova to placement
will fail with some keystoneauth1 exception, but because of our old
friend @safe_connect we likely won't fail the server delete because we
squash the exception from KSA:

https://github.com/openstack/nova/blob/0f102089dd0b27c7d35f0cbba87332414032c0a4/nova/scheduler/client/report.py#L2069

However, you'd still have allocations in placement against resource
providers (compute nodes) for instances that no longer exist, which
means you're available capacity for scheduling new requests is
diminished until those bogus allocations are purged from placement,
which will take some scripting.

In other words, not good things.

>
>> I'm definitely interested in what you do plan to do for the database
>> migration to minimize downtime.
> At this point, I'm thinking turn off placement, setup the new one, do
> the migration
> of the placement-specific tables (this can be a straightforward documented task
> OR it would be awesome if it was a placement command (something along
> the lines of `placement-manage db import_from_nova`) which would import all
> the right things

You wouldn't also stop nova-api while doing this? Otherwise you're going
to get into the data/resource tracking mess described above which will
require some post-migration cleanup scripting.

>
> The idea of having a command would be*extremely* useful for deployment tools
> in automating the process and it also allows the placement team to selectively
> decide what they want to onboard?
>
> Just throwing ideas here.
>


--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

> I think there was a period in time where the nova_api database was created
> where entires would try to get pulled out from the original nova database and
> then checking nova_api if it doesn't exist afterwards (or vice versa). One
> of the cases that this was done to deal with was for things like instance types
> or flavours.
>
> I don't know the exact details but I know that older instance types exist in
> the nova db and the newer ones are sitting in nova_api. Something along
> those lines?

Yep, we've moved entire databases before in nova with minimal disruption
to the users. Not just flavors, but several pieces of data came out of
the "main" database and into the api database transparently. It's
doable, but with placement being split to a separate
project/repo/whatever, there's not really any option for being graceful
about it in this case.

> At this point, I'm thinking turn off placement, setup the new one, do
> the migration
> of the placement-specific tables (this can be a straightforward documented task
> OR it would be awesome if it was a placement command (something along
> the lines of `placement-manage db import_from_nova`) which would import all
> the right things
>
> The idea of having a command would be *extremely* useful for deployment tools
> in automating the process and it also allows the placement team to selectively
> decide what they want to onboard?

Well, it's pretty cut-and-dried as all the tables in nova-api are either
for nova or placement, so there's not much confusion about what belongs.

I'm not sure that doing this import in python is really the most
efficient way. I agree a placement-manage command would be ideal from an
"easy button" point of view, but I think a couple lines of bash that
call mysqldump are likely to vastly outperform us doing it natively in
python. We could script exec()s of those commands from python, but.. I
think I'd rather just see that as a shell script that people can easily
alter/test on their own.

Just curious, but in your case would the service catalog entry change at
all? If you stand up the new placement in the exact same spot, it
shouldn't, but I imagine some people will have the catalog entry change
slightly (even if just because of a VIP or port change). Am I
remembering correctly that the catalog can get cached in various places
such that much of nova would need a restart to notice?

--Dan

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On Wed, Sep 5, 2018 at 12:41 PM Dan Smith <dms@danplanet.com> wrote:
>
> > I think there was a period in time where the nova_api database was created
> > where entires would try to get pulled out from the original nova database and
> > then checking nova_api if it doesn't exist afterwards (or vice versa). One
> > of the cases that this was done to deal with was for things like instance types
> > or flavours.
> >
> > I don't know the exact details but I know that older instance types exist in
> > the nova db and the newer ones are sitting in nova_api. Something along
> > those lines?
>
> Yep, we've moved entire databases before in nova with minimal disruption
> to the users. Not just flavors, but several pieces of data came out of
> the "main" database and into the api database transparently. It's
> doable, but with placement being split to a separate
> project/repo/whatever, there's not really any option for being graceful
> about it in this case.
>
> > At this point, I'm thinking turn off placement, setup the new one, do
> > the migration
> > of the placement-specific tables (this can be a straightforward documented task
> > OR it would be awesome if it was a placement command (something along
> > the lines of `placement-manage db import_from_nova`) which would import all
> > the right things
> >
> > The idea of having a command would be *extremely* useful for deployment tools
> > in automating the process and it also allows the placement team to selectively
> > decide what they want to onboard?
>
> Well, it's pretty cut-and-dried as all the tables in nova-api are either
> for nova or placement, so there's not much confusion about what belongs.
>
> I'm not sure that doing this import in python is really the most
> efficient way. I agree a placement-manage command would be ideal from an
> "easy button" point of view, but I think a couple lines of bash that
> call mysqldump are likely to vastly outperform us doing it natively in
> python. We could script exec()s of those commands from python, but.. I
> think I'd rather just see that as a shell script that people can easily
> alter/test on their own.
>
> Just curious, but in your case would the service catalog entry change at
> all? If you stand up the new placement in the exact same spot, it
> shouldn't, but I imagine some people will have the catalog entry change
> slightly (even if just because of a VIP or port change). Am I
> remembering correctly that the catalog can get cached in various places
> such that much of nova would need a restart to notice?

We already have placement in the catalog and it's behind a load balancer
so changing the backends resolves things right away, so we likely won't
be needing any restarts (and I don't think OSA will either because it uses
the same model).

> --Dan



--
Mohammed Naser — vexxhost
-----------------------------------------------------
D. 514-316-8872
D. 800-910-1726 ext. 200
E. mnaser@vexxhost.com
W. http://vexxhost.com

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators