Mailing List Archive

On Thu, 20 Sep 2018 at 16:02, Matt Riedemann <mriedemos@gmail.com> wrote:

> On 9/20/2018 4:16 AM, John Garbutt wrote:
> > Following on from the PTG discussions, I wanted to bring everyone's
> > attention to Nova's plans to deprecate ComputeCapabilitiesFilter,
> > including most of the the integration with Ironic Capabilities.
> >
> > To be specific, this is my proposal in code form:
> > https://review.openstack.org/#/c/603102/
> >
> > Once the code we propose to deprecate is removed we will stop using
> > capabilities pushed up from Ironic for 'scheduling', but we would still
> > pass capabilities request in the flavor down to Ironic (until we get
> > some standard traits and/or deploy templates sorted for things like
> UEFI).
> >
> > Functionally, we believe all use cases can be replaced by using the
> > simpler placement traits (this is more efficient than post placement
> > filtering done using capabilities):
> >
> https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/ironic-driver-traits.html
> >
> > Please note the recent addition of forbidden traits that helps improve
> > the usefulness of the above approach:
> >
> https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-forbidden-traits.html
> >
> > For example, a flavor request for GPUs >= 2 could be replaced by a
> > custom trait trait that reports if a given Ironic node has
> > CUSTOM_MORE_THAN_2_GPUS. That is a bad example (longer term we don't
> > want to use traits for this, but that is a discussion for another day)
> > but it is the example that keeps being raised in discussions on this
> topic.
> >
> > The main reason for reaching out in this email is to ask if anyone has
> > needs that the ResourceClass and Traits scheme does not currently
> > address, or can think of a problem with a transition to the newer
> approach.
>
> I left a few comments in the change, but I'm assuming as part of the
> deprecation we'd remove the filter from the default enabled_filters list
> so new installs don't automatically get warnings during scheduling?
>

+1
Good point, we totally need to do that.


> Another thing is about existing flavors configured for these
> capabilities-scoped specs. Are you saying during the deprecation we'd
> continue to use those even if the filter is disabled? In the review I
> had suggested that we add a pre-upgrade check which inspects the flavors
> and if any of these are found, we report a warning meaning those flavors
> need to be updated to use traits rather than capabilities. Would that be
> reasonable?
>

I like the idea of a warning, but there are features that have not yet
moved to traits:
https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html

There is a more general plan that will help, but its not quite ready yet:
https://review.openstack.org/#/c/504952/

As such, I think we can't get pull the plug on flavors including
capabilities and passing them to Ironic, but (after a cycle of deprecation)
I think we can now stop pushing capabilities from Ironic into Nova and
using them for placement.

Thanks,
John

On 9/25/2018 8:36 AM, John Garbutt wrote:
> Another thing is about existing flavors configured for these
> capabilities-scoped specs. Are you saying during the deprecation we'd
> continue to use those even if the filter is disabled? In the review I
> had suggested that we add a pre-upgrade check which inspects the
> flavors
> and if any of these are found, we report a warning meaning those
> flavors
> need to be updated to use traits rather than capabilities. Would
> that be
> reasonable?
>
>
> I like the idea of a warning, but there are features that have not yet
> moved to traits:
> https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html
>
> There is a more general plan that will help, but its not quite ready yet:
> https://review.openstack.org/#/c/504952/
>
> As such, I think we can't get pull the plug on flavors including
> capabilities and passing them to Ironic, but (after a cycle of
> deprecation) I think we can now stop pushing capabilities from Ironic
> into Nova and using them for placement.

Forgive my ignorance, but if traits are not on par with capabilities,
why are we deprecating the capabilities filter?

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On Tue, 25 Sep 2018 12:08:03 -0500, Matt Riedemann wrote:
> On 9/25/2018 8:36 AM, John Garbutt wrote:
>> Another thing is about existing flavors configured for these
>> capabilities-scoped specs. Are you saying during the deprecation we'd
>> continue to use those even if the filter is disabled? In the review I
>> had suggested that we add a pre-upgrade check which inspects the
>> flavors
>> and if any of these are found, we report a warning meaning those
>> flavors
>> need to be updated to use traits rather than capabilities. Would
>> that be
>> reasonable?
>>
>>
>> I like the idea of a warning, but there are features that have not yet
>> moved to traits:
>> https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html
>>
>> There is a more general plan that will help, but its not quite ready yet:
>> https://review.openstack.org/#/c/504952/
>>
>> As such, I think we can't get pull the plug on flavors including
>> capabilities and passing them to Ironic, but (after a cycle of
>> deprecation) I think we can now stop pushing capabilities from Ironic
>> into Nova and using them for placement.
>
> Forgive my ignorance, but if traits are not on par with capabilities,
> why are we deprecating the capabilities filter?

I would like to know the answer to this as well.

-melanie





_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 09/26/2018 05:48 PM, melanie witt wrote:
> On Tue, 25 Sep 2018 12:08:03 -0500, Matt Riedemann wrote:
>> On 9/25/2018 8:36 AM, John Garbutt wrote:
>>>      Another thing is about existing flavors configured for these
>>>      capabilities-scoped specs. Are you saying during the deprecation
>>> we'd
>>>      continue to use those even if the filter is disabled? In the
>>> review I
>>>      had suggested that we add a pre-upgrade check which inspects the
>>>      flavors
>>>      and if any of these are found, we report a warning meaning those
>>>      flavors
>>>      need to be updated to use traits rather than capabilities. Would
>>>      that be
>>>      reasonable?
>>>
>>>
>>> I like the idea of a warning, but there are features that have not yet
>>> moved to traits:
>>> https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html
>>>
>>>
>>> There is a more general plan that will help, but its not quite ready
>>> yet:
>>> https://review.openstack.org/#/c/504952/
>>>
>>> As such, I think we can't get pull the plug on flavors including
>>> capabilities and passing them to Ironic, but (after a cycle of
>>> deprecation) I think we can now stop pushing capabilities from Ironic
>>> into Nova and using them for placement.
>>
>> Forgive my ignorance, but if traits are not on par with capabilities,
>> why are we deprecating the capabilities filter?
>
> I would like to know the answer to this as well.

In short, traits were never designed to be key/value pairs. They are
simple strings indicating boolean capabilities.

Ironic "capabilities" are key/value metadata pairs. *Some* of those
Ironic "capabilities" are possible to create as boolean traits.

For example, you can change the boot_mode=uefi and boot_mode=bios Ironic
capabilities to be a trait called CUSTOM_BOOT_MODE_UEFI or
CUSTOM_BOOT_MODE_BIOS [1].

Other Ironic "capabilities" are not, in fact, capabilities at all.
Instead, they are just random key/value pairs that are not boolean in
nature nor do they represent a capability of the baremetal hardware.

A great example of this would be the proposed "deploy template" from
[2]. This is nothing more than abusing the placement traits API in order
to allow passthrough of instance configuration data from the nova flavor
extra spec directly into the nodes.instance_info field in the Ironic
database. It's a hack that is abusing the entire concept of the
placement traits concept, IMHO.

We should have a way *in Nova* of allowing instance configuration
key/value information to be passed through to the virt driver's spawn()
method, much the same way we provide for user_data that gets exposed
after boot to the guest instance via configdrive or the metadata service
API. What this deploy template thing is is just a hack to get around the
fact that nova doesn't have a basic way of passing through some collated
instance configuration key/value information, which is a darn shame and
I'm really kind of annoyed with myself for not noticing this sooner. :(

-jay

[1] As I've asked for in the past, it would be great to have Ironic
contributors push patches to the os-traits library for standardized
baremetal capabilities like boot modes. Please do consider contributing
there.

[2]
https://review.openstack.org/#/c/504952/16/specs/approved/deploy-templates.rst

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 9/27/2018 3:02 PM, Jay Pipes wrote:
> A great example of this would be the proposed "deploy template" from
> [2]. This is nothing more than abusing the placement traits API in order
> to allow passthrough of instance configuration data from the nova flavor
> extra spec directly into the nodes.instance_info field in the Ironic
> database. It's a hack that is abusing the entire concept of the
> placement traits concept, IMHO.
>
> We should have a way *in Nova* of allowing instance configuration
> key/value information to be passed through to the virt driver's spawn()
> method, much the same way we provide for user_data that gets exposed
> after boot to the guest instance via configdrive or the metadata service
> API. What this deploy template thing is is just a hack to get around the
> fact that nova doesn't have a basic way of passing through some collated
> instance configuration key/value information, which is a darn shame and
> I'm really kind of annoyed with myself for not noticing this sooner. :(

We talked about this in Dublin through right? We said a good thing to do
would be to have some kind of template/profile/config/whatever stored
off in glare where schema could be registered on that thing, and then
you pass a handle (ID reference) to that to nova when creating the
(baremetal) server, nova pulls it down from glare and hands it off to
the virt driver. It's just that no one is doing that work.

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On Thu, 27 Sep 2018 17:23:26 -0500, Matt Riedemann wrote:
> On 9/27/2018 3:02 PM, Jay Pipes wrote:
>> A great example of this would be the proposed "deploy template" from
>> [2]. This is nothing more than abusing the placement traits API in order
>> to allow passthrough of instance configuration data from the nova flavor
>> extra spec directly into the nodes.instance_info field in the Ironic
>> database. It's a hack that is abusing the entire concept of the
>> placement traits concept, IMHO.
>>
>> We should have a way *in Nova* of allowing instance configuration
>> key/value information to be passed through to the virt driver's spawn()
>> method, much the same way we provide for user_data that gets exposed
>> after boot to the guest instance via configdrive or the metadata service
>> API. What this deploy template thing is is just a hack to get around the
>> fact that nova doesn't have a basic way of passing through some collated
>> instance configuration key/value information, which is a darn shame and
>> I'm really kind of annoyed with myself for not noticing this sooner. :(
>
> We talked about this in Dublin through right? We said a good thing to do
> would be to have some kind of template/profile/config/whatever stored
> off in glare where schema could be registered on that thing, and then
> you pass a handle (ID reference) to that to nova when creating the
> (baremetal) server, nova pulls it down from glare and hands it off to
> the virt driver. It's just that no one is doing that work.

If I understood correctly, that discussion was around adding a way to
pass a desired hardware configuration to nova when booting an ironic
instance. And that it's something that isn't yet possible to do using
the existing ComputeCapabilitiesFilter. Someone please correct me if I'm
wrong there.

That said, I still don't understand why we are talking about deprecating
the ComputeCapabilitiesFilter if there's no supported way to replace it
yet. If boolean traits are not enough to replace it, then we need to
hold off on deprecating it, right? Would the
template/profile/config/whatever in glare approach replace what the
ComputeCapabilitiesFilter is doing or no? Sorry, I'm just not clearly
understanding this yet.

-melanie




_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On Fri, Sep 28, 2018 at 12:50 AM melanie witt <melwittt@gmail.com> wrote:

> On Thu, 27 Sep 2018 17:23:26 -0500, Matt Riedemann wrote:
> > On 9/27/2018 3:02 PM, Jay Pipes wrote:
> >> A great example of this would be the proposed "deploy template" from
> >> [2]. This is nothing more than abusing the placement traits API in order
> >> to allow passthrough of instance configuration data from the nova flavor
> >> extra spec directly into the nodes.instance_info field in the Ironic
> >> database. It's a hack that is abusing the entire concept of the
> >> placement traits concept, IMHO.
> >>
> >> We should have a way *in Nova* of allowing instance configuration
> >> key/value information to be passed through to the virt driver's spawn()
> >> method, much the same way we provide for user_data that gets exposed
> >> after boot to the guest instance via configdrive or the metadata service
> >> API. What this deploy template thing is is just a hack to get around the
> >> fact that nova doesn't have a basic way of passing through some collated
> >> instance configuration key/value information, which is a darn shame and
> >> I'm really kind of annoyed with myself for not noticing this sooner. :(
> >
> > We talked about this in Dublin through right? We said a good thing to do
> > would be to have some kind of template/profile/config/whatever stored
> > off in glare where schema could be registered on that thing, and then
> > you pass a handle (ID reference) to that to nova when creating the
> > (baremetal) server, nova pulls it down from glare and hands it off to
> > the virt driver. It's just that no one is doing that work.
>
> If I understood correctly, that discussion was around adding a way to
> pass a desired hardware configuration to nova when booting an ironic
> instance. And that it's something that isn't yet possible to do using
> the existing ComputeCapabilitiesFilter. Someone please correct me if I'm
> wrong there.
>
> That said, I still don't understand why we are talking about deprecating
> the ComputeCapabilitiesFilter if there's no supported way to replace it
> yet. If boolean traits are not enough to replace it, then we need to
> hold off on deprecating it, right? Would the
> template/profile/config/whatever in glare approach replace what the
> ComputeCapabilitiesFilter is doing or no? Sorry, I'm just not clearly
> understanding this yet.
>
>
I just feel some new traits have to be defined, like Jay said, and some
work has to be done on the Ironic side to make sure they expose them as
traits and not by the old way.
That leaves tho a question : does Ironic support custom capabilities ? If
so, that leads to Jay's point about the key/pair information that's not
intented for traits. If we all agree on the fact that traits shouldn't be
allowed for key/value pairs, could we somehow imagine Ironic to change the
customization mechanism to be boolean only ?

Also, I'm a bit confused whether operators make use of Ironic capabilities
for fancy operational queries, like the ones we have in
https://github.com/openstack/nova/blob/3716752/nova/scheduler/filters/extra_specs_ops.py#L24-L35
and if Ironic correctly documents how to put such things into traits ? (eg.
say CUSTOM_I_HAVE_MORE_THAN_2_GPUS)

All of the above makes me a bit worried by a possible
ComputeCapabilitiesFilter deprecation, if we aren't yet able to provide a
clear upgrade path for our users.

-Sylvain

-melanie
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>